2021-04-04T09:20:22.937 INFO:root:teuthology version: 1.1.0-85d61eae 2021-04-04T09:20:22.939 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/teuthology-2021-04-04_06:07:03-rbd-pacific-distro-basic-smithi/6019607 --owner scheduled_teuthology@teuthology --description rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-policy-simple} --name teuthology-2021-04-04_06:07:03-rbd-pacific-distro-basic-smithi -- /home/teuthworker/archive/teuthology-2021-04-04_06:07:03-rbd-pacific-distro-basic-smithi/6019607/orig.config.yaml 2021-04-04T09:20:22.993 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-04-04T09:20:23.055 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2021-04-04_06:07:03-rbd-pacific-distro-basic-smithi/6019607 branch: pacific description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-policy-simple} email: ceph-qa@ceph.io first_in_suite: false job_id: '6019607' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run the rbd_mirror_journal.sh workunit to test the rbd-mirror daemon name: teuthology-2021-04-04_06:07:03-rbd-pacific-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: pacific ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.4: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.5: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.6: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd objectstore: filestore osd sloppy crc: true fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: f4fb41419a42bee43a2069637ef17064073bf30b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: sha1: f4fb41419a42bee43a2069637ef17064073bf30b workunit: branch: pacific sha1: f4fb41419a42bee43a2069637ef17064073bf30b owner: scheduled_teuthology@teuthology priority: 80 repo: https://github.com/ceph/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 sha1: f4fb41419a42bee43a2069637ef17064073bf30b sleep_before_teardown: 0 suite: rbd suite_branch: pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph_f4fb41419a42bee43a2069637ef17064073bf30b/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: f4fb41419a42bee43a2069637ef17064073bf30b targets: smithi078.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCox+hBSbqhZEOOs8jJK9PcXf99OuJixxSDbvBBJ8g0Jqe3aOnAMO+buNziqMRwbKJXzwgNnWt9t7jtlfJE+H+CfWSV1mTaaI4R3SGDIdVGXHuu38Y7B1XVImZeAvGYrlvVuquCp6oLBt4bucRosTI7NuXdLkm6f6DCOQJkhV09jum2gvIT0VtQrSEAFKIDU129Cr1q9zKd+7mHZwHM2KrT0974/I2dvr5qXy+6MKfDRd5P8wwKoVtlinNbVpPZ6NVyIKm5aOa+QOE7W2jIV/ObgS34+grwZ3kDeE5GvqcI7tZErKB5M5zat6zk125M+fyQ/xSBbE6I70tQ3kqoSDC7 smithi122.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0B/RVSuPh8DWhZMw8FntzsniAdnj1e24NvMQ7NpKZhgVshkWgPW0KytLhlnNC+PusZl6HqGhurZeByzBqf8oZwNNJa03FA9WRMHEURT6LJnP54V4HePaqUTn8mv8dALIvAyKSBiRLhEc99GOpKnJhwOqnLi/QimZerlOBSQbyNgtCdsZnKCpmlHpXI6NWwK1VzlblAhCLMnGdlLzCDQW5NJ53DfbIZNCFUX3r7Xn2bHzJyYCKVkIVIMhK4h0f0X+HxlkJ0zHzPrutH8Yzl96PUPn7TyttNXI3jhPpaaFXBGxk9OvHSxBvb87eIeWbaG8fz/xuz6DxcP6MazsCcwcn tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_journal.sh env: CEPH_ARGS: '' RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' teuthology_branch: master teuthology_sha1: 85d61eae4759f46ce21e9a37cd816a7a1a66c9d5 timestamp: 2021-04-04_06:07:03 tube: smithi user: teuthology verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.13277 2021-04-04T09:20:23.056 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_f4fb41419a42bee43a2069637ef17064073bf30b/qa; will attempt to use it 2021-04-04T09:20:23.058 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_f4fb41419a42bee43a2069637ef17064073bf30b/qa/tasks 2021-04-04T09:20:23.059 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-04-04T09:20:23.061 INFO:teuthology.task.internal:Checking packages... 2021-04-04T09:20:23.121 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash 'f4fb41419a42bee43a2069637ef17064073bf30b' 2021-04-04T09:20:23.121 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-04-04T09:20:23.122 INFO:teuthology.packaging:ref: None 2021-04-04T09:20:23.122 INFO:teuthology.packaging:tag: None 2021-04-04T09:20:23.123 INFO:teuthology.packaging:branch: pacific 2021-04-04T09:20:23.123 INFO:teuthology.packaging:sha1: f4fb41419a42bee43a2069637ef17064073bf30b 2021-04-04T09:20:23.123 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=pacific 2021-04-04T09:20:23.322 INFO:teuthology.task.internal:Found packages for ceph version 16.2.0-22-gf4fb4141-1bionic 2021-04-04T09:20:23.323 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-04-04T09:20:23.330 INFO:teuthology.task.internal:no buildpackages task found 2021-04-04T09:20:23.331 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-04-04T09:20:23.377 INFO:teuthology.task.internal:Saving configuration 2021-04-04T09:20:23.395 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-04-04T09:20:23.403 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-04T09:20:23.421 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-04-04 09:14:57.860146', 'description': '/home/teuthworker/archive/teuthology-2021-04-04_06:07:03-rbd-pacific-distro-basic-smithi/6019607', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAJ/4zJdlgu62zeMNw6ZruYeensBSjjEvfAjUjMudwtV', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi078.front.sepia.ceph.com'} 2021-04-04T09:20:23.439 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-04-04 09:14:57.860128', 'description': '/home/teuthworker/archive/teuthology-2021-04-04_06:07:03-rbd-pacific-distro-basic-smithi/6019607', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAdH8FQPqTJ9zkeYu8KgaxtXJ34Cbjsw68JO5ZwbsKe6', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi122.front.sepia.ceph.com'} 2021-04-04T09:20:23.439 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-04-04T09:20:23.447 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2021-04-04T09:20:23.448 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2021-04-04T09:20:23.448 INFO:teuthology.run_tasks:Running task console_log... 2021-04-04T09:20:23.531 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f55402d7a60>, signals=[15]) 2021-04-04T09:20:23.532 INFO:teuthology.run_tasks:Running task internal.connect... 2021-04-04T09:20:23.541 INFO:teuthology.task.internal:Opening connections... 2021-04-04T09:20:23.541 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2021-04-04T09:20:23.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-04T09:20:23.617 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2021-04-04T09:20:23.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-04T09:20:23.696 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-04-04T09:20:23.707 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2021-04-04T09:20:23.816 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2021-04-04T09:20:23.817 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2021-04-04T09:20:23.865 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="Ubuntu" 2021-04-04T09:20:23.866 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-04-04T09:20:23.866 INFO:teuthology.orchestra.run.smithi078.stdout:ID=ubuntu 2021-04-04T09:20:23.866 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE=debian 2021-04-04T09:20:23.866 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-04-04T09:20:23.867 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="18.04" 2021-04-04T09:20:23.867 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://www.ubuntu.com/" 2021-04-04T09:20:23.867 INFO:teuthology.orchestra.run.smithi078.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-04-04T09:20:23.867 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-04-04T09:20:23.867 INFO:teuthology.orchestra.run.smithi078.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-04-04T09:20:23.868 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_CODENAME=bionic 2021-04-04T09:20:23.868 INFO:teuthology.orchestra.run.smithi078.stdout:UBUNTU_CODENAME=bionic 2021-04-04T09:20:23.869 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2021-04-04T09:20:23.886 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2021-04-04T09:20:23.899 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2021-04-04T09:20:23.900 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2021-04-04T09:20:23.947 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="Ubuntu" 2021-04-04T09:20:23.948 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-04-04T09:20:23.948 INFO:teuthology.orchestra.run.smithi122.stdout:ID=ubuntu 2021-04-04T09:20:23.948 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE=debian 2021-04-04T09:20:23.949 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-04-04T09:20:23.949 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="18.04" 2021-04-04T09:20:23.950 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://www.ubuntu.com/" 2021-04-04T09:20:23.950 INFO:teuthology.orchestra.run.smithi122.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-04-04T09:20:23.950 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-04-04T09:20:23.951 INFO:teuthology.orchestra.run.smithi122.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-04-04T09:20:23.952 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_CODENAME=bionic 2021-04-04T09:20:23.952 INFO:teuthology.orchestra.run.smithi122.stdout:UBUNTU_CODENAME=bionic 2021-04-04T09:20:23.952 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2021-04-04T09:20:23.980 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-04-04T09:20:23.992 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-04-04T09:20:24.014 INFO:teuthology.task.internal:Checking for old test directory... 2021-04-04T09:20:24.014 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2021-04-04T09:20:24.017 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2021-04-04T09:20:24.024 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-04-04T09:20:24.034 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-04-04T09:20:24.035 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2021-04-04T09:20:24.065 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2021-04-04T09:20:24.084 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-04-04T09:20:24.151 INFO:teuthology.run_tasks:Running task kernel... 2021-04-04T09:20:24.168 DEBUG:teuthology.task.kernel:normalized config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'cluster1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'cluster1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'cluster1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.0': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.0': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'cluster2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'cluster2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.mirror': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.mirror.0': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.mirror.1': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.mirror.2': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.mirror.3': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.mirror.4': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.mirror.5': {'kdb': True, 'sha1': 'distro'}, 'cluster1.client.mirror.6': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.mirror': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.mirror.0': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.mirror.1': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.mirror.2': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.mirror.3': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.mirror.4': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.mirror.5': {'kdb': True, 'sha1': 'distro'}, 'cluster2.client.mirror.6': {'kdb': True, 'sha1': 'distro'}} 2021-04-04T09:20:24.168 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-04-04T09:20:24.169 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2021-04-04T09:20:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:4.15.0-136-generic 2021-04-04T09:20:24.176 INFO:teuthology.task.kernel:Running kernel on smithi078: 4.15.0-136-generic 2021-04-04T09:20:24.176 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get clean 2021-04-04T09:20:24.289 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get update 2021-04-04T09:20:24.406 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-04-04T09:20:24.532 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-04-04T09:20:24.612 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-04-04T09:20:24.663 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,639 kB] 2021-04-04T09:20:24.769 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [950 kB] 2021-04-04T09:20:24.785 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [310 kB] 2021-04-04T09:20:24.788 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [17.7 kB] 2021-04-04T09:20:24.788 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [265 kB] 2021-04-04T09:20:24.792 INFO:teuthology.orchestra.run.smithi078.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [34.6 kB] 2021-04-04T09:20:24.793 INFO:teuthology.orchestra.run.smithi078.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,118 kB] 2021-04-04T09:20:24.802 INFO:teuthology.orchestra.run.smithi078.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [980 kB] 2021-04-04T09:20:24.811 INFO:teuthology.orchestra.run.smithi078.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [251 kB] 2021-04-04T09:20:24.814 INFO:teuthology.orchestra.run.smithi078.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,480 B] 2021-04-04T09:20:24.814 INFO:teuthology.orchestra.run.smithi078.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.1 kB] 2021-04-04T09:20:24.815 INFO:teuthology.orchestra.run.smithi078.stdout:Get:15 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [4,412 B] 2021-04-04T09:20:24.918 INFO:teuthology.orchestra.run.smithi078.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-04-04T09:20:25.054 INFO:teuthology.orchestra.run.smithi078.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,253 kB] 2021-04-04T09:20:25.346 INFO:teuthology.orchestra.run.smithi078.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,978 kB] 2021-04-04T09:20:25.473 INFO:teuthology.orchestra.run.smithi078.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [402 kB] 2021-04-04T09:20:25.486 INFO:teuthology.orchestra.run.smithi078.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [24.3 kB] 2021-04-04T09:20:25.486 INFO:teuthology.orchestra.run.smithi078.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [286 kB] 2021-04-04T09:20:25.522 INFO:teuthology.orchestra.run.smithi078.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [38.2 kB] 2021-04-04T09:20:25.523 INFO:teuthology.orchestra.run.smithi078.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,562 kB] 2021-04-04T09:20:25.584 INFO:teuthology.orchestra.run.smithi078.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,724 kB] 2021-04-04T09:20:25.616 INFO:teuthology.orchestra.run.smithi078.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [365 kB] 2021-04-04T09:20:25.623 INFO:teuthology.orchestra.run.smithi078.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [24.9 kB] 2021-04-04T09:20:25.624 INFO:teuthology.orchestra.run.smithi078.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.6 kB] 2021-04-04T09:20:25.624 INFO:teuthology.orchestra.run.smithi078.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,464 B] 2021-04-04T09:20:26.611 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 13.5 MB in 2s (6,446 kB/s) 2021-04-04T09:20:27.987 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2021-04-04T09:20:28.036 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-04-04T09:20:28.113 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2021-04-04T09:20:28.370 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2021-04-04T09:20:28.371 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2021-04-04T09:20:28.454 INFO:teuthology.orchestra.run.smithi078.stderr:E: Unable to locate package linux-image-current-generic 2021-04-04T09:20:28.455 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-04-04T09:20:28.456 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-04-04T09:20:28.534 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2021-04-04T09:20:28.794 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2021-04-04T09:20:28.794 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2021-04-04T09:20:28.972 INFO:teuthology.orchestra.run.smithi078.stdout:The following additional packages will be installed: 2021-04-04T09:20:28.972 INFO:teuthology.orchestra.run.smithi078.stdout: linux-generic linux-headers-4.15.0-140 linux-headers-4.15.0-140-generic 2021-04-04T09:20:28.972 INFO:teuthology.orchestra.run.smithi078.stdout: linux-headers-generic linux-image-4.15.0-140-generic 2021-04-04T09:20:28.973 INFO:teuthology.orchestra.run.smithi078.stdout: linux-modules-4.15.0-140-generic linux-modules-extra-4.15.0-140-generic 2021-04-04T09:20:28.974 INFO:teuthology.orchestra.run.smithi078.stdout:Suggested packages: 2021-04-04T09:20:28.974 INFO:teuthology.orchestra.run.smithi078.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2021-04-04T09:20:28.974 INFO:teuthology.orchestra.run.smithi078.stdout:Recommended packages: 2021-04-04T09:20:28.975 INFO:teuthology.orchestra.run.smithi078.stdout: thermald 2021-04-04T09:20:29.009 INFO:teuthology.orchestra.run.smithi078.stdout:The following NEW packages will be installed: 2021-04-04T09:20:29.009 INFO:teuthology.orchestra.run.smithi078.stdout: linux-headers-4.15.0-140 linux-headers-4.15.0-140-generic 2021-04-04T09:20:29.009 INFO:teuthology.orchestra.run.smithi078.stdout: linux-image-4.15.0-140-generic linux-modules-4.15.0-140-generic 2021-04-04T09:20:29.010 INFO:teuthology.orchestra.run.smithi078.stdout: linux-modules-extra-4.15.0-140-generic 2021-04-04T09:20:29.010 INFO:teuthology.orchestra.run.smithi078.stdout:The following packages will be upgraded: 2021-04-04T09:20:29.011 INFO:teuthology.orchestra.run.smithi078.stdout: linux-generic linux-headers-generic linux-image-generic 2021-04-04T09:20:29.210 INFO:teuthology.orchestra.run.smithi078.stdout:3 upgraded, 5 newly installed, 0 to remove and 36 not upgraded. 2021-04-04T09:20:29.211 INFO:teuthology.orchestra.run.smithi078.stdout:Need to get 66.0 MB of archives. 2021-04-04T09:20:29.211 INFO:teuthology.orchestra.run.smithi078.stdout:After this operation, 334 MB of additional disk space will be used. 2021-04-04T09:20:29.211 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-140-generic amd64 4.15.0-140.144 [13.1 MB] 2021-04-04T09:20:30.491 INFO:teuthology.orchestra.run.smithi078.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-140-generic amd64 4.15.0-140.144 [8,021 kB] 2021-04-04T09:20:30.744 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-140-generic amd64 4.15.0-140.144 [32.8 MB] 2021-04-04T09:20:31.878 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.140.127 [1,868 B] 2021-04-04T09:20:31.879 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.140.127 [2,580 B] 2021-04-04T09:20:31.879 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-140 all 4.15.0-140.144 [10.9 MB] 2021-04-04T09:20:32.198 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-140-generic amd64 4.15.0-140.144 [1,101 kB] 2021-04-04T09:20:32.214 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.140.127 [2,484 B] 2021-04-04T09:20:32.748 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 66.0 MB in 3s (20.7 MB/s) 2021-04-04T09:20:32.910 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-modules-4.15.0-140-generic. 2021-04-04T09:20:32.946 INFO:teuthology.orchestra.run.smithi078.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102065 files and directories currently installed.) 2021-04-04T09:20:32.949 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../0-linux-modules-4.15.0-140-generic_4.15.0-140.144_amd64.deb ... 2021-04-04T09:20:32.998 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-modules-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:20:34.705 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-image-4.15.0-140-generic. 2021-04-04T09:20:34.717 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../1-linux-image-4.15.0-140-generic_4.15.0-140.144_amd64.deb ... 2021-04-04T09:20:34.806 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-image-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:20:35.333 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-140-generic. 2021-04-04T09:20:35.344 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-140-generic_4.15.0-140.144_amd64.deb ... 2021-04-04T09:20:35.383 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-modules-extra-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:20:39.129 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../3-linux-generic_4.15.0.140.127_amd64.deb ... 2021-04-04T09:20:39.258 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-generic (4.15.0.140.127) over (4.15.0.136.123) ... 2021-04-04T09:20:39.572 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.140.127_amd64.deb ... 2021-04-04T09:20:39.678 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-image-generic (4.15.0.140.127) over (4.15.0.136.123) ... 2021-04-04T09:20:40.005 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-headers-4.15.0-140. 2021-04-04T09:20:40.019 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../5-linux-headers-4.15.0-140_4.15.0-140.144_all.deb ... 2021-04-04T09:20:40.064 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-headers-4.15.0-140 (4.15.0-140.144) ... 2021-04-04T09:20:44.695 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-headers-4.15.0-140-generic. 2021-04-04T09:20:44.716 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../6-linux-headers-4.15.0-140-generic_4.15.0-140.144_amd64.deb ... 2021-04-04T09:20:44.753 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-headers-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:20:47.166 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.140.127_amd64.deb ... 2021-04-04T09:20:47.287 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-headers-generic (4.15.0.140.127) over (4.15.0.136.123) ... 2021-04-04T09:20:47.639 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-headers-4.15.0-140 (4.15.0-140.144) ... 2021-04-04T09:20:47.778 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-modules-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:20:48.302 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-image-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:20:48.758 INFO:teuthology.orchestra.run.smithi078.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-140-generic 2021-04-04T09:20:48.758 INFO:teuthology.orchestra.run.smithi078.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-140-generic 2021-04-04T09:20:48.887 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-modules-extra-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:20:49.389 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-image-generic (4.15.0.140.127) ... 2021-04-04T09:20:49.524 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-headers-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:20:49.650 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-headers-generic (4.15.0.140.127) ... 2021-04-04T09:20:49.776 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-generic (4.15.0.140.127) ... 2021-04-04T09:20:49.893 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for linux-image-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:20:50.104 INFO:teuthology.orchestra.run.smithi078.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-04-04T09:20:50.105 INFO:teuthology.orchestra.run.smithi078.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-140-generic 2021-04-04T09:21:05.217 INFO:teuthology.orchestra.run.smithi078.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-04-04T09:21:05.218 INFO:teuthology.orchestra.run.smithi078.stdout:Sourcing file `/etc/default/grub' 2021-04-04T09:21:05.253 INFO:teuthology.orchestra.run.smithi078.stdout:Generating grub configuration file ... 2021-04-04T09:21:05.592 INFO:teuthology.orchestra.run.smithi078.stdout:Found linux image: /boot/vmlinuz-4.15.0-140-generic 2021-04-04T09:21:05.627 INFO:teuthology.orchestra.run.smithi078.stdout:Found initrd image: /boot/initrd.img-4.15.0-140-generic 2021-04-04T09:21:05.940 INFO:teuthology.orchestra.run.smithi078.stdout:Found linux image: /boot/vmlinuz-4.15.0-136-generic 2021-04-04T09:21:05.974 INFO:teuthology.orchestra.run.smithi078.stdout:Found initrd image: /boot/initrd.img-4.15.0-136-generic 2021-04-04T09:21:07.410 INFO:teuthology.orchestra.run.smithi078.stdout:done 2021-04-04T09:21:07.796 DEBUG:teuthology.orchestra.run.smithi078:> dpkg -s linux-image-generic 2021-04-04T09:21:07.815 INFO:teuthology.orchestra.run.smithi078.stdout:Package: linux-image-generic 2021-04-04T09:21:07.816 INFO:teuthology.orchestra.run.smithi078.stdout:Status: install ok installed 2021-04-04T09:21:07.816 INFO:teuthology.orchestra.run.smithi078.stdout:Priority: optional 2021-04-04T09:21:07.816 INFO:teuthology.orchestra.run.smithi078.stdout:Section: kernel 2021-04-04T09:21:07.817 INFO:teuthology.orchestra.run.smithi078.stdout:Installed-Size: 17 2021-04-04T09:21:07.817 INFO:teuthology.orchestra.run.smithi078.stdout:Maintainer: Ubuntu Kernel Team 2021-04-04T09:21:07.817 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: amd64 2021-04-04T09:21:07.817 INFO:teuthology.orchestra.run.smithi078.stdout:Source: linux-meta 2021-04-04T09:21:07.818 INFO:teuthology.orchestra.run.smithi078.stdout:Version: 4.15.0.140.127 2021-04-04T09:21:07.818 INFO:teuthology.orchestra.run.smithi078.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-04-04T09:21:07.818 INFO:teuthology.orchestra.run.smithi078.stdout:Depends: linux-image-4.15.0-140-generic, linux-modules-extra-4.15.0-140-generic, linux-firmware, intel-microcode, amd64-microcode 2021-04-04T09:21:07.818 INFO:teuthology.orchestra.run.smithi078.stdout:Recommends: thermald 2021-04-04T09:21:07.818 INFO:teuthology.orchestra.run.smithi078.stdout:Description: Generic Linux kernel image 2021-04-04T09:21:07.819 INFO:teuthology.orchestra.run.smithi078.stdout: This package will always depend on the latest generic kernel image 2021-04-04T09:21:07.819 INFO:teuthology.orchestra.run.smithi078.stdout: available. 2021-04-04T09:21:07.819 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-136-generic Expected: 4.15.0-140-generic 2021-04-04T09:21:07.820 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2021-04-04T09:21:07.827 INFO:teuthology.orchestra.run.smithi122.stdout:4.15.0-136-generic 2021-04-04T09:21:07.827 INFO:teuthology.task.kernel:Running kernel on smithi122: 4.15.0-136-generic 2021-04-04T09:21:07.828 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get clean 2021-04-04T09:21:07.947 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get update 2021-04-04T09:21:08.080 INFO:teuthology.orchestra.run.smithi122.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-04-04T09:21:08.207 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-04-04T09:21:08.304 INFO:teuthology.orchestra.run.smithi122.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-04-04T09:21:08.311 INFO:teuthology.orchestra.run.smithi122.stdout:Get:4 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,639 kB] 2021-04-04T09:21:08.421 INFO:teuthology.orchestra.run.smithi122.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [950 kB] 2021-04-04T09:21:08.437 INFO:teuthology.orchestra.run.smithi122.stdout:Get:6 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [310 kB] 2021-04-04T09:21:08.440 INFO:teuthology.orchestra.run.smithi122.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [265 kB] 2021-04-04T09:21:08.443 INFO:teuthology.orchestra.run.smithi122.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [17.7 kB] 2021-04-04T09:21:08.443 INFO:teuthology.orchestra.run.smithi122.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [34.6 kB] 2021-04-04T09:21:08.444 INFO:teuthology.orchestra.run.smithi122.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,118 kB] 2021-04-04T09:21:08.454 INFO:teuthology.orchestra.run.smithi122.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [980 kB] 2021-04-04T09:21:08.463 INFO:teuthology.orchestra.run.smithi122.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [251 kB] 2021-04-04T09:21:08.466 INFO:teuthology.orchestra.run.smithi122.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,480 B] 2021-04-04T09:21:08.466 INFO:teuthology.orchestra.run.smithi122.stdout:Get:14 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.1 kB] 2021-04-04T09:21:08.466 INFO:teuthology.orchestra.run.smithi122.stdout:Get:15 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [4,412 B] 2021-04-04T09:21:08.599 INFO:teuthology.orchestra.run.smithi122.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-04-04T09:21:08.765 INFO:teuthology.orchestra.run.smithi122.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,253 kB] 2021-04-04T09:21:09.138 INFO:teuthology.orchestra.run.smithi122.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,978 kB] 2021-04-04T09:21:09.260 INFO:teuthology.orchestra.run.smithi122.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [402 kB] 2021-04-04T09:21:09.271 INFO:teuthology.orchestra.run.smithi122.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [286 kB] 2021-04-04T09:21:09.279 INFO:teuthology.orchestra.run.smithi122.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [24.3 kB] 2021-04-04T09:21:09.280 INFO:teuthology.orchestra.run.smithi122.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [38.2 kB] 2021-04-04T09:21:09.281 INFO:teuthology.orchestra.run.smithi122.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,724 kB] 2021-04-04T09:21:09.332 INFO:teuthology.orchestra.run.smithi122.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,562 kB] 2021-04-04T09:21:09.377 INFO:teuthology.orchestra.run.smithi122.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [365 kB] 2021-04-04T09:21:09.383 INFO:teuthology.orchestra.run.smithi122.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [24.9 kB] 2021-04-04T09:21:09.384 INFO:teuthology.orchestra.run.smithi122.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.6 kB] 2021-04-04T09:21:09.385 INFO:teuthology.orchestra.run.smithi122.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,464 B] 2021-04-04T09:21:10.271 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 13.5 MB in 2s (6,455 kB/s) 2021-04-04T09:21:11.671 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2021-04-04T09:21:11.716 DEBUG:teuthology.orchestra.run.smithi122:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-04-04T09:21:11.791 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2021-04-04T09:21:12.024 INFO:teuthology.orchestra.run.smithi122.stdout:Building dependency tree... 2021-04-04T09:21:12.025 INFO:teuthology.orchestra.run.smithi122.stdout:Reading state information... 2021-04-04T09:21:12.105 INFO:teuthology.orchestra.run.smithi122.stderr:E: Unable to locate package linux-image-current-generic 2021-04-04T09:21:12.107 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-04-04T09:21:12.108 DEBUG:teuthology.orchestra.run.smithi122:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-04-04T09:21:12.226 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2021-04-04T09:21:12.457 INFO:teuthology.orchestra.run.smithi122.stdout:Building dependency tree... 2021-04-04T09:21:12.458 INFO:teuthology.orchestra.run.smithi122.stdout:Reading state information... 2021-04-04T09:21:12.635 INFO:teuthology.orchestra.run.smithi122.stdout:The following additional packages will be installed: 2021-04-04T09:21:12.636 INFO:teuthology.orchestra.run.smithi122.stdout: linux-generic linux-headers-4.15.0-140 linux-headers-4.15.0-140-generic 2021-04-04T09:21:12.636 INFO:teuthology.orchestra.run.smithi122.stdout: linux-headers-generic linux-image-4.15.0-140-generic 2021-04-04T09:21:12.636 INFO:teuthology.orchestra.run.smithi122.stdout: linux-modules-4.15.0-140-generic linux-modules-extra-4.15.0-140-generic 2021-04-04T09:21:12.637 INFO:teuthology.orchestra.run.smithi122.stdout:Suggested packages: 2021-04-04T09:21:12.638 INFO:teuthology.orchestra.run.smithi122.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2021-04-04T09:21:12.638 INFO:teuthology.orchestra.run.smithi122.stdout:Recommended packages: 2021-04-04T09:21:12.638 INFO:teuthology.orchestra.run.smithi122.stdout: thermald 2021-04-04T09:21:12.670 INFO:teuthology.orchestra.run.smithi122.stdout:The following NEW packages will be installed: 2021-04-04T09:21:12.671 INFO:teuthology.orchestra.run.smithi122.stdout: linux-headers-4.15.0-140 linux-headers-4.15.0-140-generic 2021-04-04T09:21:12.671 INFO:teuthology.orchestra.run.smithi122.stdout: linux-image-4.15.0-140-generic linux-modules-4.15.0-140-generic 2021-04-04T09:21:12.671 INFO:teuthology.orchestra.run.smithi122.stdout: linux-modules-extra-4.15.0-140-generic 2021-04-04T09:21:12.672 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages will be upgraded: 2021-04-04T09:21:12.673 INFO:teuthology.orchestra.run.smithi122.stdout: linux-generic linux-headers-generic linux-image-generic 2021-04-04T09:21:12.854 INFO:teuthology.orchestra.run.smithi122.stdout:3 upgraded, 5 newly installed, 0 to remove and 36 not upgraded. 2021-04-04T09:21:12.854 INFO:teuthology.orchestra.run.smithi122.stdout:Need to get 66.0 MB of archives. 2021-04-04T09:21:12.855 INFO:teuthology.orchestra.run.smithi122.stdout:After this operation, 334 MB of additional disk space will be used. 2021-04-04T09:21:12.855 INFO:teuthology.orchestra.run.smithi122.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-140-generic amd64 4.15.0-140.144 [13.1 MB] 2021-04-04T09:21:13.856 INFO:teuthology.orchestra.run.smithi122.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-140-generic amd64 4.15.0-140.144 [8,021 kB] 2021-04-04T09:21:14.099 INFO:teuthology.orchestra.run.smithi122.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-140-generic amd64 4.15.0-140.144 [32.8 MB] 2021-04-04T09:21:15.094 INFO:teuthology.orchestra.run.smithi122.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.140.127 [1,868 B] 2021-04-04T09:21:15.095 INFO:teuthology.orchestra.run.smithi122.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.140.127 [2,580 B] 2021-04-04T09:21:15.095 INFO:teuthology.orchestra.run.smithi122.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-140 all 4.15.0-140.144 [10.9 MB] 2021-04-04T09:21:15.384 INFO:teuthology.orchestra.run.smithi122.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-140-generic amd64 4.15.0-140.144 [1,101 kB] 2021-04-04T09:21:15.421 INFO:teuthology.orchestra.run.smithi122.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.140.127 [2,484 B] 2021-04-04T09:21:15.848 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 66.0 MB in 3s (24.1 MB/s) 2021-04-04T09:21:15.982 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-modules-4.15.0-140-generic. 2021-04-04T09:21:16.017 INFO:teuthology.orchestra.run.smithi122.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102065 files and directories currently installed.) 2021-04-04T09:21:16.020 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../0-linux-modules-4.15.0-140-generic_4.15.0-140.144_amd64.deb ... 2021-04-04T09:21:16.070 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-modules-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:21:17.604 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-image-4.15.0-140-generic. 2021-04-04T09:21:17.617 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../1-linux-image-4.15.0-140-generic_4.15.0-140.144_amd64.deb ... 2021-04-04T09:21:17.707 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-image-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:21:18.223 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-140-generic. 2021-04-04T09:21:18.235 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-140-generic_4.15.0-140.144_amd64.deb ... 2021-04-04T09:21:18.273 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-modules-extra-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:21:22.170 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../3-linux-generic_4.15.0.140.127_amd64.deb ... 2021-04-04T09:21:22.291 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-generic (4.15.0.140.127) over (4.15.0.136.123) ... 2021-04-04T09:21:22.611 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.140.127_amd64.deb ... 2021-04-04T09:21:22.736 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-image-generic (4.15.0.140.127) over (4.15.0.136.123) ... 2021-04-04T09:21:23.039 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-headers-4.15.0-140. 2021-04-04T09:21:23.052 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../5-linux-headers-4.15.0-140_4.15.0-140.144_all.deb ... 2021-04-04T09:21:23.089 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-headers-4.15.0-140 (4.15.0-140.144) ... 2021-04-04T09:21:27.643 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package linux-headers-4.15.0-140-generic. 2021-04-04T09:21:27.665 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../6-linux-headers-4.15.0-140-generic_4.15.0-140.144_amd64.deb ... 2021-04-04T09:21:27.710 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-headers-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:21:30.120 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.140.127_amd64.deb ... 2021-04-04T09:21:30.229 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking linux-headers-generic (4.15.0.140.127) over (4.15.0.136.123) ... 2021-04-04T09:21:30.607 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-headers-4.15.0-140 (4.15.0-140.144) ... 2021-04-04T09:21:30.775 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-modules-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:21:31.344 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-image-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:21:31.775 INFO:teuthology.orchestra.run.smithi122.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-140-generic 2021-04-04T09:21:31.776 INFO:teuthology.orchestra.run.smithi122.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-140-generic 2021-04-04T09:21:31.913 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-modules-extra-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:21:32.407 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-image-generic (4.15.0.140.127) ... 2021-04-04T09:21:32.533 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-headers-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:21:32.659 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-headers-generic (4.15.0.140.127) ... 2021-04-04T09:21:32.785 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up linux-generic (4.15.0.140.127) ... 2021-04-04T09:21:32.895 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for linux-image-4.15.0-140-generic (4.15.0-140.144) ... 2021-04-04T09:21:33.094 INFO:teuthology.orchestra.run.smithi122.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-04-04T09:21:33.094 INFO:teuthology.orchestra.run.smithi122.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-140-generic 2021-04-04T09:21:48.294 INFO:teuthology.orchestra.run.smithi122.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-04-04T09:21:48.294 INFO:teuthology.orchestra.run.smithi122.stdout:Sourcing file `/etc/default/grub' 2021-04-04T09:21:48.327 INFO:teuthology.orchestra.run.smithi122.stdout:Generating grub configuration file ... 2021-04-04T09:21:48.684 INFO:teuthology.orchestra.run.smithi122.stdout:Found linux image: /boot/vmlinuz-4.15.0-140-generic 2021-04-04T09:21:48.735 INFO:teuthology.orchestra.run.smithi122.stdout:Found initrd image: /boot/initrd.img-4.15.0-140-generic 2021-04-04T09:21:49.037 INFO:teuthology.orchestra.run.smithi122.stdout:Found linux image: /boot/vmlinuz-4.15.0-136-generic 2021-04-04T09:21:49.070 INFO:teuthology.orchestra.run.smithi122.stdout:Found initrd image: /boot/initrd.img-4.15.0-136-generic 2021-04-04T09:21:50.035 INFO:teuthology.orchestra.run.smithi122.stdout:done 2021-04-04T09:21:50.423 DEBUG:teuthology.orchestra.run.smithi122:> dpkg -s linux-image-generic 2021-04-04T09:21:50.445 INFO:teuthology.orchestra.run.smithi122.stdout:Package: linux-image-generic 2021-04-04T09:21:50.446 INFO:teuthology.orchestra.run.smithi122.stdout:Status: install ok installed 2021-04-04T09:21:50.446 INFO:teuthology.orchestra.run.smithi122.stdout:Priority: optional 2021-04-04T09:21:50.446 INFO:teuthology.orchestra.run.smithi122.stdout:Section: kernel 2021-04-04T09:21:50.447 INFO:teuthology.orchestra.run.smithi122.stdout:Installed-Size: 17 2021-04-04T09:21:50.447 INFO:teuthology.orchestra.run.smithi122.stdout:Maintainer: Ubuntu Kernel Team 2021-04-04T09:21:50.448 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: amd64 2021-04-04T09:21:50.448 INFO:teuthology.orchestra.run.smithi122.stdout:Source: linux-meta 2021-04-04T09:21:50.448 INFO:teuthology.orchestra.run.smithi122.stdout:Version: 4.15.0.140.127 2021-04-04T09:21:50.449 INFO:teuthology.orchestra.run.smithi122.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-04-04T09:21:50.450 INFO:teuthology.orchestra.run.smithi122.stdout:Depends: linux-image-4.15.0-140-generic, linux-modules-extra-4.15.0-140-generic, linux-firmware, intel-microcode, amd64-microcode 2021-04-04T09:21:50.450 INFO:teuthology.orchestra.run.smithi122.stdout:Recommends: thermald 2021-04-04T09:21:50.450 INFO:teuthology.orchestra.run.smithi122.stdout:Description: Generic Linux kernel image 2021-04-04T09:21:50.451 INFO:teuthology.orchestra.run.smithi122.stdout: This package will always depend on the latest generic kernel image 2021-04-04T09:21:50.451 INFO:teuthology.orchestra.run.smithi122.stdout: available. 2021-04-04T09:21:50.452 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-136-generic Expected: 4.15.0-140-generic 2021-04-04T09:21:50.453 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-04-04T09:21:50.454 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-04-04T09:21:50.454 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-04-04T09:21:50.455 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2021-04-04T09:21:50.455 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2021-04-04T09:21:50.456 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get clean 2021-04-04T09:21:50.520 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get update 2021-04-04T09:21:50.770 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-04-04T09:21:50.770 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:2 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-04-04T09:21:50.839 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-04-04T09:21:50.919 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-04-04T09:21:52.668 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2021-04-04T09:21:52.716 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-04-04T09:21:52.796 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2021-04-04T09:21:53.054 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2021-04-04T09:21:53.055 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2021-04-04T09:21:53.136 INFO:teuthology.orchestra.run.smithi078.stderr:E: Unable to locate package linux-image-current-generic 2021-04-04T09:21:53.137 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-04-04T09:21:53.138 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-04-04T09:21:53.216 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2021-04-04T09:21:53.470 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2021-04-04T09:21:53.471 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2021-04-04T09:21:53.673 INFO:teuthology.orchestra.run.smithi078.stdout:linux-image-generic is already the newest version (4.15.0.140.127). 2021-04-04T09:21:53.674 INFO:teuthology.orchestra.run.smithi078.stdout:0 upgraded, 0 newly installed, 0 to remove and 36 not upgraded. 2021-04-04T09:21:53.676 DEBUG:teuthology.orchestra.run.smithi078:> dpkg -s linux-image-generic 2021-04-04T09:21:53.699 INFO:teuthology.orchestra.run.smithi078.stdout:Package: linux-image-generic 2021-04-04T09:21:53.700 INFO:teuthology.orchestra.run.smithi078.stdout:Status: install ok installed 2021-04-04T09:21:53.700 INFO:teuthology.orchestra.run.smithi078.stdout:Priority: optional 2021-04-04T09:21:53.701 INFO:teuthology.orchestra.run.smithi078.stdout:Section: kernel 2021-04-04T09:21:53.701 INFO:teuthology.orchestra.run.smithi078.stdout:Installed-Size: 17 2021-04-04T09:21:53.701 INFO:teuthology.orchestra.run.smithi078.stdout:Maintainer: Ubuntu Kernel Team 2021-04-04T09:21:53.702 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: amd64 2021-04-04T09:21:53.702 INFO:teuthology.orchestra.run.smithi078.stdout:Source: linux-meta 2021-04-04T09:21:53.702 INFO:teuthology.orchestra.run.smithi078.stdout:Version: 4.15.0.140.127 2021-04-04T09:21:53.702 INFO:teuthology.orchestra.run.smithi078.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-04-04T09:21:53.703 INFO:teuthology.orchestra.run.smithi078.stdout:Depends: linux-image-4.15.0-140-generic, linux-modules-extra-4.15.0-140-generic, linux-firmware, intel-microcode, amd64-microcode 2021-04-04T09:21:53.703 INFO:teuthology.orchestra.run.smithi078.stdout:Recommends: thermald 2021-04-04T09:21:53.703 INFO:teuthology.orchestra.run.smithi078.stdout:Description: Generic Linux kernel image 2021-04-04T09:21:53.703 INFO:teuthology.orchestra.run.smithi078.stdout: This package will always depend on the latest generic kernel image 2021-04-04T09:21:53.704 INFO:teuthology.orchestra.run.smithi078.stdout: available. 2021-04-04T09:21:53.704 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2021-04-04T09:21:53.753 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.AFGinlVs3M 2021-04-04T09:21:53.754 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub/grub.cfg /tmp/tmp.AFGinlVs3M 2021-04-04T09:21:53.809 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.AFGinlVs3M 2021-04-04T09:21:53.929 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.AFGinlVs3M is 8KB 2021-04-04T09:21:53.985 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.AFGinlVs3M 2021-04-04T09:21:53.993 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-04-04T09:21:54.049 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:21:54.049 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-04-04T09:21:54.050 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-04-04T09:21:54.136 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-140-generic 2021-04-04T09:21:54.136 DEBUG:teuthology.orchestra.run.smithi078:> sudo update-grub 2021-04-04T09:21:55.408 INFO:teuthology.orchestra.run.smithi078.stderr:Sourcing file `/etc/default/grub' 2021-04-04T09:21:55.429 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2021-04-04T09:21:55.708 INFO:teuthology.orchestra.run.smithi078.stderr:Found linux image: /boot/vmlinuz-4.15.0-140-generic 2021-04-04T09:21:55.720 INFO:teuthology.orchestra.run.smithi078.stderr:Found initrd image: /boot/initrd.img-4.15.0-140-generic 2021-04-04T09:21:56.048 INFO:teuthology.orchestra.run.smithi078.stderr:Found linux image: /boot/vmlinuz-4.15.0-136-generic 2021-04-04T09:21:56.062 INFO:teuthology.orchestra.run.smithi078.stderr:Found initrd image: /boot/initrd.img-4.15.0-136-generic 2021-04-04T09:21:57.189 INFO:teuthology.orchestra.run.smithi078.stderr:done 2021-04-04T09:21:57.191 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2021-04-04T09:21:57.195 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2021-04-04T09:21:57.196 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=None, version=distro) 2021-04-04T09:21:57.197 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get clean 2021-04-04T09:21:57.275 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get update 2021-04-04T09:21:57.408 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-04-04T09:21:57.527 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-04-04T09:21:57.603 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-04-04T09:21:57.683 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-04-04T09:21:59.281 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2021-04-04T09:21:59.328 DEBUG:teuthology.orchestra.run.smithi122:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-04-04T09:21:59.405 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2021-04-04T09:21:59.652 INFO:teuthology.orchestra.run.smithi122.stdout:Building dependency tree... 2021-04-04T09:21:59.653 INFO:teuthology.orchestra.run.smithi122.stdout:Reading state information... 2021-04-04T09:21:59.734 INFO:teuthology.orchestra.run.smithi122.stderr:E: Unable to locate package linux-image-current-generic 2021-04-04T09:21:59.736 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-04-04T09:21:59.737 DEBUG:teuthology.orchestra.run.smithi122:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-04-04T09:21:59.817 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2021-04-04T09:22:00.078 INFO:teuthology.orchestra.run.smithi122.stdout:Building dependency tree... 2021-04-04T09:22:00.079 INFO:teuthology.orchestra.run.smithi122.stdout:Reading state information... 2021-04-04T09:22:00.280 INFO:teuthology.orchestra.run.smithi122.stdout:linux-image-generic is already the newest version (4.15.0.140.127). 2021-04-04T09:22:00.280 INFO:teuthology.orchestra.run.smithi122.stdout:0 upgraded, 0 newly installed, 0 to remove and 36 not upgraded. 2021-04-04T09:22:00.282 DEBUG:teuthology.orchestra.run.smithi122:> dpkg -s linux-image-generic 2021-04-04T09:22:00.303 INFO:teuthology.orchestra.run.smithi122.stdout:Package: linux-image-generic 2021-04-04T09:22:00.304 INFO:teuthology.orchestra.run.smithi122.stdout:Status: install ok installed 2021-04-04T09:22:00.304 INFO:teuthology.orchestra.run.smithi122.stdout:Priority: optional 2021-04-04T09:22:00.304 INFO:teuthology.orchestra.run.smithi122.stdout:Section: kernel 2021-04-04T09:22:00.304 INFO:teuthology.orchestra.run.smithi122.stdout:Installed-Size: 17 2021-04-04T09:22:00.305 INFO:teuthology.orchestra.run.smithi122.stdout:Maintainer: Ubuntu Kernel Team 2021-04-04T09:22:00.305 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: amd64 2021-04-04T09:22:00.305 INFO:teuthology.orchestra.run.smithi122.stdout:Source: linux-meta 2021-04-04T09:22:00.305 INFO:teuthology.orchestra.run.smithi122.stdout:Version: 4.15.0.140.127 2021-04-04T09:22:00.305 INFO:teuthology.orchestra.run.smithi122.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-04-04T09:22:00.306 INFO:teuthology.orchestra.run.smithi122.stdout:Depends: linux-image-4.15.0-140-generic, linux-modules-extra-4.15.0-140-generic, linux-firmware, intel-microcode, amd64-microcode 2021-04-04T09:22:00.306 INFO:teuthology.orchestra.run.smithi122.stdout:Recommends: thermald 2021-04-04T09:22:00.306 INFO:teuthology.orchestra.run.smithi122.stdout:Description: Generic Linux kernel image 2021-04-04T09:22:00.306 INFO:teuthology.orchestra.run.smithi122.stdout: This package will always depend on the latest generic kernel image 2021-04-04T09:22:00.307 INFO:teuthology.orchestra.run.smithi122.stdout: available. 2021-04-04T09:22:00.307 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2021-04-04T09:22:00.357 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.qwWZqhgIbg 2021-04-04T09:22:00.358 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub/grub.cfg /tmp/tmp.qwWZqhgIbg 2021-04-04T09:22:00.413 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.qwWZqhgIbg 2021-04-04T09:22:00.531 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.qwWZqhgIbg is 8KB 2021-04-04T09:22:00.583 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.qwWZqhgIbg 2021-04-04T09:22:00.594 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-04-04T09:22:00.650 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:22:00.651 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-04-04T09:22:00.651 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-04-04T09:22:00.730 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-140-generic 2021-04-04T09:22:00.731 DEBUG:teuthology.orchestra.run.smithi122:> sudo update-grub 2021-04-04T09:22:02.081 INFO:teuthology.orchestra.run.smithi122.stderr:Sourcing file `/etc/default/grub' 2021-04-04T09:22:02.108 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2021-04-04T09:22:02.356 INFO:teuthology.orchestra.run.smithi122.stderr:Found linux image: /boot/vmlinuz-4.15.0-140-generic 2021-04-04T09:22:02.369 INFO:teuthology.orchestra.run.smithi122.stderr:Found initrd image: /boot/initrd.img-4.15.0-140-generic 2021-04-04T09:22:02.675 INFO:teuthology.orchestra.run.smithi122.stderr:Found linux image: /boot/vmlinuz-4.15.0-136-generic 2021-04-04T09:22:02.688 INFO:teuthology.orchestra.run.smithi122.stderr:Found initrd image: /boot/initrd.img-4.15.0-136-generic 2021-04-04T09:22:03.321 INFO:teuthology.orchestra.run.smithi122.stderr:done 2021-04-04T09:22:03.323 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2021-04-04T09:22:33.356 INFO:teuthology.misc:Re-opening connections... 2021-04-04T09:22:33.356 INFO:teuthology.misc:trying to connect to ubuntu@smithi078.front.sepia.ceph.com 2021-04-04T09:22:33.357 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-04T09:22:33.358 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-04T09:22:51.867 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2021-04-04T09:22:52.112 INFO:teuthology.misc:trying to connect to ubuntu@smithi122.front.sepia.ceph.com 2021-04-04T09:22:52.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-04T09:22:52.114 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-04T09:23:52.175 DEBUG:teuthology.orchestra.remote:timed out 2021-04-04T09:23:52.180 DEBUG:teuthology.misc:waited 78.8236632347107 2021-04-04T09:23:53.182 INFO:teuthology.misc:trying to connect to ubuntu@smithi078.front.sepia.ceph.com 2021-04-04T09:23:53.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-04T09:23:53.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-04T09:23:53.264 DEBUG:teuthology.orchestra.run.smithi078:> true 2021-04-04T09:23:53.605 DEBUG:teuthology.misc:waited 80.24812078475952 2021-04-04T09:23:54.606 INFO:teuthology.misc:trying to connect to ubuntu@smithi122.front.sepia.ceph.com 2021-04-04T09:23:54.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-04T09:23:54.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-04T09:23:54.764 DEBUG:teuthology.orchestra.run.smithi122:> true 2021-04-04T09:23:55.084 DEBUG:teuthology.misc:waited 81.72746467590332 2021-04-04T09:23:56.085 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2021-04-04T09:23:56.085 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "4.15.0-140-generic"... 2021-04-04T09:23:56.086 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2021-04-04T09:23:56.092 INFO:teuthology.orchestra.run.smithi078.stdout:4.15.0-140-generic 2021-04-04T09:23:56.093 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-140-generic vs 4.15.0-140-generic 2021-04-04T09:23:56.093 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-04-04T09:23:56.094 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-04-04T09:23:56.094 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2021-04-04T09:23:56.094 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "4.15.0-140-generic"... 2021-04-04T09:23:56.095 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2021-04-04T09:23:56.101 INFO:teuthology.orchestra.run.smithi122.stdout:4.15.0-140-generic 2021-04-04T09:23:56.101 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-140-generic vs 4.15.0-140-generic 2021-04-04T09:23:56.102 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-04-04T09:23:56.102 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-04-04T09:23:57.103 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2021-04-04T09:23:57.104 DEBUG:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-04-04T09:23:57.133 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2021-04-04T09:23:57.169 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2021-04-04T09:23:57.170 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-04-04T09:23:57.292 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2021-04-04T09:23:57.316 INFO:teuthology.run_tasks:Running task internal.base... 2021-04-04T09:23:57.326 INFO:teuthology.task.internal:Creating test directory... 2021-04-04T09:23:57.328 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-04-04T09:23:57.330 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-04-04T09:23:57.337 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-04-04T09:23:57.350 INFO:teuthology.run_tasks:Running task internal.archive... 2021-04-04T09:23:57.361 INFO:teuthology.task.internal:Creating archive directory... 2021-04-04T09:23:57.361 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-04-04T09:23:57.380 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-04-04T09:23:57.409 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-04-04T09:23:57.420 INFO:teuthology.task.internal:Enabling coredump saving... 2021-04-04T09:23:57.420 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-04-04T09:23:57.444 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-04-04T09:23:57.463 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-04-04T09:23:57.471 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-04-04T09:23:57.477 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-04-04T09:23:57.485 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-04-04T09:23:57.486 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-04-04T09:23:57.495 INFO:teuthology.task.internal:Configuring sudo... 2021-04-04T09:23:57.496 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-04-04T09:23:57.516 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-04-04T09:23:57.543 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-04-04T09:23:57.557 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-04-04T09:23:57.558 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-04-04T09:23:57.573 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-04-04T09:23:57.592 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-04-04T09:23:57.625 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-04-04T09:23:57.673 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:23:57.674 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-04-04T09:23:57.732 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-04-04T09:23:57.739 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-04-04T09:23:57.788 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:23:57.789 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-04-04T09:23:57.849 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2021-04-04T09:23:57.852 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2021-04-04T09:23:57.931 INFO:teuthology.run_tasks:Running task internal.timer... 2021-04-04T09:23:57.940 INFO:teuthology.task.internal:Starting timer... 2021-04-04T09:23:57.941 INFO:teuthology.run_tasks:Running task pcp... 2021-04-04T09:23:57.960 INFO:teuthology.run_tasks:Running task selinux... 2021-04-04T09:23:57.982 DEBUG:teuthology.task.selinux:Excluding smithi078: OS 'ubuntu' does not support SELinux 2021-04-04T09:23:57.983 DEBUG:teuthology.task.selinux:Excluding smithi122: OS 'ubuntu' does not support SELinux 2021-04-04T09:23:57.984 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-04-04T09:23:57.984 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-04-04T09:23:57.984 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-04-04T09:23:57.984 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-04-04T09:23:58.002 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2021-04-04T09:23:58.025 INFO:teuthology.repo_utils:Fetching master from origin 2021-04-04T09:23:58.186 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-04-04T09:23:58.321 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-04-04T09:23:58.328 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi122.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-04-04T09:23:59.036 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-04-04T09:24:00.689 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-04-04T09:24:03.014 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-04-04T09:24:03.064 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:03 +0000 (0:00:00.082) 0:00:00.082 ********** 2021-04-04T09:24:03.080 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:03 +0000 (0:00:00.016) 0:00:00.098 ********** 2021-04-04T09:24:04.779 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-04-04T09:24:04.781 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi122.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi122.front.sepia.ceph.com] 2021-04-04T09:24:04.789 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi078.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi078.front.sepia.ceph.com] 2021-04-04T09:24:04.853 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:04 +0000 (0:00:01.773) 0:00:01.871 ********** 2021-04-04T09:24:04.872 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:04 +0000 (0:00:00.018) 0:00:01.890 ********** 2021-04-04T09:24:05.097 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-04-04T09:24:05.099 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-04-04T09:24:05.121 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:05 +0000 (0:00:00.248) 0:00:02.139 ********** 2021-04-04T09:24:05.127 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-04-04T09:24:05.154 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:05 +0000 (0:00:00.032) 0:00:02.171 ********** 2021-04-04T09:24:05.771 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-04-04T09:24:05.773 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-04-04T09:24:05.795 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:05 +0000 (0:00:00.641) 0:00:02.813 ********** 2021-04-04T09:24:05.977 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-04-04T09:24:06.000 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:05 +0000 (0:00:00.205) 0:00:03.018 ********** 2021-04-04T09:24:06.182 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-04-04T09:24:06.183 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.065586", "end": "2021-04-04 09:24:06.158380", "rc": 0, "start": "2021-04-04 09:24:06.092794", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-04-04T09:24:06.204 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:06 +0000 (0:00:00.203) 0:00:03.222 ********** 2021-04-04T09:24:06.252 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.077319", "end": "2021-04-04 09:24:06.630300", "rc": 0, "start": "2021-04-04 09:24:06.552981", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-04-04T09:24:06.274 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:06 +0000 (0:00:00.069) 0:00:03.292 ********** 2021-04-04T09:24:06.477 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-04-04T09:24:06.479 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi122.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-04T09:24:06.503 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:06 +0000 (0:00:00.228) 0:00:03.521 ********** 2021-04-04T09:24:06.522 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:06 +0000 (0:00:00.018) 0:00:03.539 ********** 2021-04-04T09:24:07.354 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-04-04T09:24:07.356 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1617528246.5556028-14892-244131439408192/source", "state": "file", "uid": 0} changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1617528246.5359957-14890-76871111789776/source", "state": "file", "uid": 0} 2021-04-04T09:24:07.378 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:07 +0000 (0:00:00.856) 0:00:04.396 ********** 2021-04-04T09:24:07.401 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:07 +0000 (0:00:00.022) 0:00:04.418 ********** 2021-04-04T09:24:07.856 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-04-04T09:24:07.858 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-04-04T09:24:07.859 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-04-04T09:24:07.879 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-04-04T09:24:07.881 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-04-04T09:24:07.958 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-04-04T09:24:07.961 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-04-04T09:24:08.117 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:08 +0000 (0:00:00.716) 0:00:05.135 ********** 2021-04-04T09:24:08.137 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:08 +0000 (0:00:00.019) 0:00:05.154 ********** 2021-04-04T09:24:08.812 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi122.front.sepia.ceph.com] 2021-04-04T09:24:08.880 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-04-04T09:24:08.882 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:08 +0000 (0:00:00.743) 0:00:05.897 ********** 2021-04-04T09:24:08.894 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] 2021-04-04T09:24:08.973 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:08 +0000 (0:00:00.092) 0:00:05.990 ********** 2021-04-04T09:24:09.762 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-04-04T09:24:09.764 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.517961", "end": "2021-04-04 09:24:09.726349", "rc": 0, "start": "2021-04-04 09:24:09.208388", "stderr": "", "stderr_lines": [], "stdout": "5bb5aa76bc3fd5b5b3f6df845aec3bd1f9062232", "stdout_lines": ["5bb5aa76bc3fd5b5b3f6df845aec3bd1f9062232"]} changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.569212", "end": "2021-04-04 09:24:09.730976", "rc": 0, "start": "2021-04-04 09:24:09.161764", "stderr": "", "stderr_lines": [], "stdout": "5bb5aa76bc3fd5b5b3f6df845aec3bd1f9062232", "stdout_lines": ["5bb5aa76bc3fd5b5b3f6df845aec3bd1f9062232"]} 2021-04-04T09:24:09.786 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:09 +0000 (0:00:00.813) 0:00:06.803 ********** 2021-04-04T09:24:09.806 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:09 +0000 (0:00:00.019) 0:00:06.823 ********** 2021-04-04T09:24:09.936 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-04-04T09:24:09.937 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.015278", "end": "2021-04-04 09:24:09.911331", "failed_when_result": false, "rc": 0, "start": "2021-04-04 09:24:09.896053", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-04-04T09:24:09.962 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:09 +0000 (0:00:00.156) 0:00:06.980 ********** 2021-04-04T09:24:10.030 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-04-04T09:24:10.032 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:10.053 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:10 +0000 (0:00:00.091) 0:00:07.071 ********** 2021-04-04T09:24:10.111 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-04-04T09:24:10.113 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.172004", "end": "2021-04-04 09:24:10.486515", "failed_when_result": false, "rc": 0, "start": "2021-04-04 09:24:10.314511", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-04-04T09:24:10.134 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:10 +0000 (0:00:00.080) 0:00:07.152 ********** 2021-04-04T09:24:10.139 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-04-04T09:24:10.141 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:10.170 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:10 +0000 (0:00:00.035) 0:00:07.187 ********** 2021-04-04T09:24:10.207 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-04-04T09:24:10.209 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:10.231 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:10 +0000 (0:00:00.061) 0:00:07.249 ********** 2021-04-04T09:24:10.236 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-04-04T09:24:10.238 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:10.274 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:10 +0000 (0:00:00.043) 0:00:07.292 ********** 2021-04-04T09:24:10.314 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-04-04T09:24:10.315 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:10.339 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:10 +0000 (0:00:00.064) 0:00:07.357 ********** 2021-04-04T09:24:10.355 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-04-04T09:24:10.357 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:10.385 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:10 +0000 (0:00:00.046) 0:00:07.403 ********** 2021-04-04T09:24:10.433 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-04-04T09:24:10.435 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:10.458 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:10 +0000 (0:00:00.072) 0:00:07.475 ********** 2021-04-04T09:24:10.463 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-04-04T09:24:10.465 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:10.489 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:10 +0000 (0:00:00.031) 0:00:07.507 ********** 2021-04-04T09:24:10.516 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-04-04T09:24:10.517 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:10.540 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:10 +0000 (0:00:00.051) 0:00:07.558 ********** 2021-04-04T09:24:10.609 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-04-04T09:24:10.611 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:10.634 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:10 +0000 (0:00:00.093) 0:00:07.651 ********** 2021-04-04T09:24:10.683 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-04-04T09:24:10.685 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-04T09:24:10.708 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:10 +0000 (0:00:00.074) 0:00:07.726 ********** 2021-04-04T09:24:10.841 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-04T09:24:10.866 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:10 +0000 (0:00:00.157) 0:00:07.884 ********** 2021-04-04T09:24:10.967 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-04-04T09:24:10.969 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-04-04T09:24:11.105 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-04-04T09:24:11.106 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-04-04T09:24:11.254 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-04-04T09:24:11.277 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-04-04T09:24:11.393 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-04-04T09:24:11.415 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-04-04T09:24:11.541 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-04-04T09:24:11.585 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-04-04T09:24:11.681 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-04-04T09:24:11.745 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-04-04T09:24:11.830 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-04-04T09:24:11.883 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-04-04T09:24:11.966 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-04-04T09:24:12.051 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-04-04T09:24:12.116 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-04-04T09:24:12.191 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-04-04T09:24:12.255 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-04-04T09:24:12.351 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-04-04T09:24:12.394 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-04-04T09:24:12.522 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-04-04T09:24:12.545 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-04-04T09:24:12.677 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-04-04T09:24:12.686 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-04-04T09:24:12.833 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-04-04T09:24:12.835 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-04-04T09:24:12.973 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-04-04T09:24:13.006 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-04-04T09:24:13.112 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-04-04T09:24:13.134 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-04-04T09:24:13.260 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-04-04T09:24:13.304 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-04-04T09:24:13.401 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-04-04T09:24:13.455 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-04-04T09:24:13.541 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-04-04T09:24:13.617 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-04-04T09:24:13.693 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-04-04T09:24:13.768 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-04-04T09:24:13.846 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-04-04T09:24:14.235 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "appen 2021-04-04T09:24:14.238 INFO:teuthology.task.ansible.out:d": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-04-04T09:24:14.252 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-04-04T09:24:14.318 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-04-04T09:24:14.408 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-04-04T09:24:14.491 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-04-04T09:24:14.567 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-04-04T09:24:14.674 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-04-04T09:24:14.728 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-04-04T09:24:14.888 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-04-04T09:24:14.891 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-04-04T09:24:15.042 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-04-04T09:24:15.054 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-04-04T09:24:15.203 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-04-04T09:24:15.236 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-04-04T09:24:15.365 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-04-04T09:24:15.418 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-04-04T09:24:15.515 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-04-04T09:24:15.580 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-04-04T09:24:15.665 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-04-04T09:24:15.740 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-04-04T09:24:15.805 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-04-04T09:24:15.902 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-04-04T09:24:15.946 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-04-04T09:24:16.052 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-04-04T09:24:16.085 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-04-04T09:24:16.191 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-04-04T09:24:16.234 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-04-04T09:24:16.349 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-04-04T09:24:16.382 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-04-04T09:24:16.498 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-04-04T09:24:16.521 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-04-04T09:24:16.657 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-04-04T09:24:16.658 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-04-04T09:24:16.806 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-04-04T09:24:16.818 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-04-04T09:24:16.956 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-04-04T09:24:16.958 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-04-04T09:24:17.099 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-04-04T09:24:17.133 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-04-04T09:24:17.238 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-04-04T09:24:17.280 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-04-04T09:24:17.395 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-04-04T09:24:17.448 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-04-04T09:24:17.532 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-04-04T09:24:17.596 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-04-04T09:24:17.671 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-04-04T09:24:17.767 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-04-04T09:24:17.820 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-04-04T09:24:17.906 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-04-04T09:24:17.959 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-04-04T09:24:18.077 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-04-04T09:24:18.110 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-04-04T09:24:18.206 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-04-04T09:24:18.260 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-04-04T09:24:18.377 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-04-04T09:24:18.401 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-04-04T09:24:18.548 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-04-04T09:24:18.550 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-04-04T09:24:18.688 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-04-04T09:24:18.701 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-04-04T09:24:18.839 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-04-04T09:24:18.841 INFO:teuthology.task.ansible.out: 2021-04-04T09:24:18.862 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-04-04T09:24:18.864 INFO:teuthology.task.ansible.out: 2021-04-04T09:24:18.980 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-04-04T09:24:19.013 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-04-04T09:24:19.129 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-04-04T09:24:19.173 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-04-04T09:24:19.279 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-04-04T09:24:19.344 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-04-04T09:24:19.431 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-04-04T09:24:19.496 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-04-04T09:24:19.570 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-04-04T09:24:19.646 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-04-04T09:24:19.711 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-04-04T09:24:19.797 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-04-04T09:24:19.861 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-04-04T09:24:19.956 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-04-04T09:24:20.000 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-04-04T09:24:20.115 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-04-04T09:24:20.149 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-04-04T09:24:20.277 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-04-04T09:24:20.288 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-04-04T09:24:20.416 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-04-04T09:24:20.439 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-04-04T09:24:20.566 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-04-04T09:24:20.578 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-04-04T09:24:20.727 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-04-04T09:24:20.729 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-04-04T09:24:20.867 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-04-04T09:24:20.869 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-04-04T09:24:21.009 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-04-04T09:24:21.010 INFO:teuthology.task.ansible.out: 2021-04-04T09:24:21.139 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-04-04T09:24:21.167 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-04-04T09:24:21.318 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-04-04T09:24:21.426 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-04-04T09:24:21.501 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-04-04T09:24:21.597 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-04-04T09:24:21.662 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-04-04T09:24:21.738 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-04-04T09:24:21.835 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-04-04T09:24:21.889 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-04-04T09:24:21.987 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-04-04T09:24:22.029 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-04-04T09:24:22.155 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-04-04T09:24:22.178 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-04-04T09:24:22.326 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-04-04T09:24:22.327 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-04-04T09:24:22.482 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-04-04T09:24:22.493 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-04-04T09:24:22.624 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-04-04T09:24:22.657 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-04-04T09:24:22.775 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-04-04T09:24:22.797 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-04-04T09:24:22.915 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-04-04T09:24:22.917 INFO:teuthology.task.ansible.out: 2021-04-04T09:24:22.959 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-04-04T09:24:23.067 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-04-04T09:24:23.131 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-04-04T09:24:23.206 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-04-04T09:24:23.293 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-04-04T09:24:23.358 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-04-04T09:24:23.454 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-04-04T09:24:23.510 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-04-04T09:24:23.606 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-04-04T09:24:23.661 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-04-04T09:24:23.779 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-04-04T09:24:23.824 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-04-04T09:24:23.940 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-04-04T09:24:23.962 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-04-04T09:24:24.100 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-04-04T09:24:24.101 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-04-04T09:24:24.250 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-04-04T09:24:24.273 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-04-04T09:24:24.394 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-04-04T09:24:24.427 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-04-04T09:24:24.535 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-04-04T09:24:24.590 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-04-04T09:24:24.685 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-04-04T09:24:24.749 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-04-04T09:24:24.835 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-04-04T09:24:24.921 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-04-04T09:24:24.977 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-04-04T09:24:25.084 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-04-04T09:24:25.128 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-04-04T09:24:25.261 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-04-04T09:24:25.263 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-04-04T09:24:25.413 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-04-04T09:24:25.436 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-04-04T09:24:25.563 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-04-04T09:24:25.585 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-04-04T09:24:25.701 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-04-04T09:24:25.755 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-04-04T09:24:25.850 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-04-04T09:24:25.915 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-04-04T09:24:26.002 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-04-04T09:24:26.079 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-04-04T09:24:26.145 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-04-04T09:24:26.242 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-04-04T09:24:26.286 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-04-04T09:24:26.382 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-04-04T09:24:26.426 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-04-04T09:24:26.544 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-04-04T09:24:26.567 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-04-04T09:24:26.706 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-04-04T09:24:26.708 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-04-04T09:24:26.857 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-04-04T09:24:26.869 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-04-04T09:24:27.007 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-04-04T09:24:27.029 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-04-04T09:24:27.158 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-04-04T09:24:27.181 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-04-04T09:24:27.298 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-04-04T09:24:27.332 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-04-04T09:24:27.439 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-04-04T09:24:27.482 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-04-04T09:24:27.579 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-04-04T09:24:27.646 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-04-04T09:24:27.733 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-04-04T09:24:27.787 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-04-04T09:24:27.883 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-04-04T09:24:27.959 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-04-04T09:24:28.024 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-04-04T09:24:28.100 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-04-04T09:24:28.166 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-04-04T09:24:28.274 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-04-04T09:24:28.318 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-04-04T09:24:28.434 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-04-04T09:24:28.467 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-04-04T09:24:28.595 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-04-04T09:24:28.606 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-04-04T09:24:28.752 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-04-04T09:24:28.774 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-04-04T09:24:28.889 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-04-04T09:24:28.945 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-04-04T09:24:29.041 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-04-04T09:24:29.094 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-04-04T09:24:29.180 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-04-04T09:24:29.261 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-04-04T09:24:29.315 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-04-04T09:24:29.421 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-04-04T09:24:29.464 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-04-04T09:24:29.582 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-04-04T09:24:29.605 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-04-04T09:24:29.742 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-04-04T09:24:29.856 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-04-04T09:24:29.858 INFO:teuthology.task.ansible.out: 2021-04-04T09:24:29.890 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-04-04T09:24:30.006 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-04-04T09:24:30.038 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-04-04T09:24:30.145 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-04-04T09:24:30.211 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-04-04T09:24:30.285 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-04-04T09:24:30.350 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-04-04T09:24:30.436 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-04-04T09:24:30.523 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-04-04T09:24:30.577 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-04-04T09:24:30.685 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-04-04T09:24:30.730 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-04-04T09:24:30.858 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-04-04T09:24:30.882 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-04-04T09:24:31.021 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-04-04T09:24:31.033 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-04-04T09:24:31.171 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-04-04T09:24:31.204 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-04-04T09:24:31.323 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-04-04T09:24:31.366 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-04-04T09:24:31.514 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-04-04T09:24:31.662 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-04-04T09:24:31.831 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-04-04T09:24:31.885 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "name": "dachary", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-04-04T09:24:32.085 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-04-04T09:24:32.109 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:32 +0000 (0:00:21.243) 0:00:29.127 ********** 2021-04-04T09:24:32.131 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-04-04T09:24:32.184 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-04-04T09:24:32.212 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:32 +0000 (0:00:00.103) 0:00:29.230 ********** 2021-04-04T09:24:32.295 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-04-04T09:24:32.297 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-04-04T09:24:32.298 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-04-04T09:24:32.300 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-04-04T09:24:32.302 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-04-04T09:24:32.305 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-04-04T09:24:32.306 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-04-04T09:24:32.309 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de 2021-04-04T09:24:32.310 INFO:teuthology.task.ansible.out:53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e8 2021-04-04T09:24:32.312 INFO:teuthology.task.ansible.out:16f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5d 2021-04-04T09:24:32.312 INFO:teuthology.task.ansible.out:e374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30 2021-04-04T09:24:32.314 INFO:teuthology.task.ansible.out:d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyC 2021-04-04T09:24:32.315 INFO:teuthology.task.ansible.out:pUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca 2021-04-04T09:24:32.317 INFO:teuthology.task.ansible.out:5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38 2021-04-04T09:24:32.318 INFO:teuthology.task.ansible.out:d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@l 2021-04-04T09:24:32.320 INFO:teuthology.task.ansible.out:aptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, 2021-04-04T09:24:32.322 INFO:teuthology.task.ansible.out: {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thin 2021-04-04T09:24:32.324 INFO:teuthology.task.ansible.out:kpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}]}, "changed": false} 2021-04-04T09:24:32.350 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:32 +0000 (0:00:00.137) 0:00:29.368 ********** 2021-04-04T09:24:32.440 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-04-04T09:24:32.442 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-04-04T09:24:32.589 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-04-04T09:24:32.738 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-04-04T09:24:32.907 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-04-04T09:24:33.076 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-04-04T09:24:33.215 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-04-04T09:24:33.452 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-04-04T09:24:33.453 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"after": "5bb5aa76bc3fd5b5b3f6df845aec3bd1f9062232", "attempts": 1, "before": "5bb5aa76bc3fd5b5b3f6df845aec3bd1f9062232", "changed": false, "remote_url_changed": false} 2021-04-04T09:24:33.477 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:33 +0000 (0:00:01.126) 0:00:30.494 ********** 2021-04-04T09:24:33.725 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-04-04T09:24:33.727 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-04-04T09:24:33.865 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-04-04T09:24:33.867 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** changed: [smithi122.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "name": "dachary", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-04-04T09:24:34.019 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:34 +0000 (0:00:00.542) 0:00:31.037 ********** 2021-04-04T09:24:34.025 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-04-04T09:24:34.027 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-04-04T09:24:34.094 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-04-04T09:24:34.118 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:34 +0000 (0:00:00.099) 0:00:31.136 ********** 2021-04-04T09:24:34.137 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-04-04T09:24:34.139 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-04-04T09:24:34.141 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-04-04T09:24:34.205 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-04-04T09:24:34.207 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-04-04T09:24:34.209 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-04-04T09:24:34.211 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-04-04T09:24:34.213 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-04-04T09:24:34.215 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de 2021-04-04T09:24:34.217 INFO:teuthology.task.ansible.out:53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e8 2021-04-04T09:24:34.218 INFO:teuthology.task.ansible.out:16f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30 2021-04-04T09:24:34.220 INFO:teuthology.task.ansible.out:d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyC 2021-04-04T09:24:34.222 INFO:teuthology.task.ansible.out:pUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca 2021-04-04T09:24:34.224 INFO:teuthology.task.ansible.out:5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38 2021-04-04T09:24:34.225 INFO:teuthology.task.ansible.out:d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullste 2021-04-04T09:24:34.227 INFO:teuthology.task.ansible.out:am 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f 2021-04-04T09:24:34.229 INFO:teuthology.task.ansible.out:9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, 2021-04-04T09:24:34.229 INFO:teuthology.task.ansible.out:{"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}]}, "changed": false} 2021-04-04T09:24:34.258 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:34 +0000 (0:00:00.139) 0:00:31.276 ********** 2021-04-04T09:24:34.275 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-04-04T09:24:34.277 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-04-04T09:24:34.403 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-04-04T09:24:34.405 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-04-04T09:24:34.529 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-04-04T09:24:34.657 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-04-04T09:24:34.794 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-04-04T09:24:34.805 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:34.900 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-04-04T09:24:34.902 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"after": "5bb5aa76bc3fd5b5b3f6df845aec3bd1f9062232", "attempts": 1, "before": "5bb5aa76bc3fd5b5b3f6df845aec3bd1f9062232", "changed": false, "remote_url_changed": false} 2021-04-04T09:24:34.923 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:34 +0000 (0:00:00.665) 0:00:31.941 ********** 2021-04-04T09:24:34.939 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-04-04T09:24:34.941 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-04-04T09:24:34.943 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-04-04T09:24:35.066 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-04-04T09:24:35.068 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-04-04T09:24:35.078 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:35.079 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:35.173 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-04-04T09:24:35.216 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-04-04T09:24:35.312 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-04-04T09:24:35.344 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-04-04T09:24:35.346 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-04-04T09:24:35.461 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-04-04T09:24:35.472 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-04-04T09:24:35.590 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-04-04T09:24:35.592 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-04-04T09:24:35.613 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-04-04T09:24:35.614 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-04-04T09:24:35.718 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-04-04T09:24:35.740 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-04-04T09:24:35.864 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-04-04T09:24:35.866 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-04-04T09:24:35.875 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-04-04T09:24:36.003 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-04-04T09:24:36.015 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-04-04T09:24:36.141 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-04-04T09:24:36.152 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-04-04T09:24:36.154 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-04-04T09:24:36.279 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-04-04T09:24:36.291 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:36.293 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-04-04T09:24:36.294 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-04-04T09:24:36.430 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-04-04T09:24:36.432 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-04-04T09:24:36.559 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-04-04T09:24:36.581 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-04-04T09:24:36.582 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-04-04T09:24:36.592 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:36.603 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:36.698 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-04-04T09:24:36.699 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-04-04T09:24:36.751 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-04-04T09:24:36.846 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-04-04T09:24:36.889 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-04-04T09:24:36.891 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-04-04T09:24:36.985 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-04-04T09:24:37.018 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-04-04T09:24:37.135 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-04-04T09:24:37.146 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-04-04T09:24:37.148 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-04-04T09:24:37.274 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-04-04T09:24:37.279 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-04-04T09:24:37.405 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-04-04T09:24:37.422 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-04-04T09:24:37.561 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-04-04T09:24:37.562 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-04-04T09:24:37.564 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-04-04T09:24:37.689 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-04-04T09:24:37.691 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-04-04T09:24:37.701 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-04-04T09:24:37.837 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-04-04T09:24:37.839 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-04-04T09:24:37.842 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-04-04T09:24:37.964 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-04-04T09:24:37.975 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-04-04T09:24:38.102 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-04-04T09:24:38.124 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-04-04T09:24:38.239 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-04-04T09:24:38.282 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-04-04T09:24:38.284 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-04-04T09:24:38.378 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-04-04T09:24:38.441 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-04-04T09:24:38.526 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-04-04T09:24:38.591 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-04-04T09:24:38.655 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-04-04T09:24:38.718 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-04-04T09:24:38.792 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-04-04T09:24:38.794 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-04-04T09:24:38.856 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-04-04T09:24:38.930 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-04-04T09:24:38.932 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-04-04T09:24:38.974 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-04-04T09:24:39.059 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-04-04T09:24:39.103 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-04-04T09:24:39.104 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-04-04T09:24:39.199 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-04-04T09:24:39.253 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-04-04T09:24:39.338 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-04-04T09:24:39.381 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-04-04T09:24:39.476 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-04-04T09:24:39.478 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-04-04T09:24:39.529 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-04-04T09:24:39.604 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-04-04T09:24:39.606 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-04-04T09:24:39.608 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-04-04T09:24:39.689 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-04-04T09:24:39.754 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-04-04T09:24:39.828 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-04-04T09:24:39.892 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-04-04T09:24:39.977 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-04-04T09:24:40.031 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-04-04T09:24:40.126 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-04-04T09:24:40.170 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-04-04T09:24:40.255 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-04-04T09:24:40.309 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-04-04T09:24:40.414 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-04-04T09:24:40.416 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-04-04T09:24:40.447 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-04-04T09:24:40.449 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-04-04T09:24:40.542 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-04-04T09:24:40.545 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-04-04T09:24:40.585 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-04-04T09:24:40.681 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-04-04T09:24:40.714 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-04-04T09:24:40.820 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-04-04T09:24:40.853 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-04-04T09:24:40.855 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-04-04T09:24:40.969 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-04-04T09:24:40.981 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-04-04T09:24:41.108 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-04-04T09:24:41.110 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-04-04T09:24:41.120 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-04-04T09:24:41.269 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-04-04T09:24:41.271 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-04-04T09:24:41.281 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-04-04T09:24:41.282 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-04-04T09:24:41.284 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-04-04T09:24:41.407 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-04-04T09:24:41.429 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-04-04T09:24:41.534 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-04-04T09:24:41.536 INFO:teuthology.task.ansible.out: 2021-04-04T09:24:41.546 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-04-04T09:24:41.670 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-04-04T09:24:41.712 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-04-04T09:24:41.796 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-04-04T09:24:41.839 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-04-04T09:24:41.965 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-04-04T09:24:41.977 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-04-04T09:24:42.101 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-04-04T09:24:42.103 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-04-04T09:24:42.112 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-04-04T09:24:42.226 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-04-04T09:24:42.248 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-04-04T09:24:42.380 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-04-04T09:24:42.381 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-04-04T09:24:42.518 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-04-04T09:24:42.520 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-04-04T09:24:42.649 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-04-04T09:24:42.670 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-04-04T09:24:42.776 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-04-04T09:24:42.828 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-04-04T09:24:42.913 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-04-04T09:24:42.966 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-04-04T09:24:42.968 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-04-04T09:24:43.040 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-04-04T09:24:43.114 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-04-04T09:24:43.178 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-04-04T09:24:43.273 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-04-04T09:24:43.315 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-04-04T09:24:43.413 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-04-04T09:24:43.455 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-04-04T09:24:43.529 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-04-04T09:24:43.582 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-04-04T09:24:43.667 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-04-04T09:24:43.720 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-04-04T09:24:43.804 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-04-04T09:24:43.847 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-04-04T09:24:43.942 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-04-04T09:24:43.984 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-04-04T09:24:44.083 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-04-04T09:24:44.085 INFO:teuthology.task.ansible.out: 2021-04-04T09:24:44.117 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-04-04T09:24:44.118 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-04-04T09:24:44.212 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-04-04T09:24:44.266 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-04-04T09:24:44.351 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-04-04T09:24:44.406 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-04-04T09:24:44.503 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-04-04T09:24:44.548 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-04-04T09:24:44.656 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-04-04T09:24:44.700 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-04-04T09:24:44.809 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-04-04T09:24:44.839 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-04-04T09:24:45.199 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansib 2021-04-04T09:24:45.200 INFO:teuthology.task.ansible.out:le_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHc 2021-04-04T09:24:45.202 INFO:teuthology.task.ansible.out:Wz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABA 2021-04-04T09:24:45.204 INFO:teuthology.task.ansible.out:AABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-04-04T09:24:45.319 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-04-04T09:24:45.327 INFO:teuthology.task.ansible.out: 2021-04-04T09:24:45.352 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-04-04T09:24:45.489 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-04-04T09:24:45.531 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-04-04T09:24:45.666 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-04-04T09:24:45.699 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-04-04T09:24:45.803 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-04-04T09:24:45.867 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-04-04T09:24:45.962 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-04-04T09:24:46.037 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-04-04T09:24:46.133 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-04-04T09:24:46.135 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-04-04T09:24:46.198 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-04-04T09:24:46.306 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-04-04T09:24:46.350 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-04-04T09:24:46.352 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-04-04T09:24:46.456 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-04-04T09:24:46.499 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-04-04T09:24:46.605 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-04-04T09:24:46.638 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-04-04T09:24:46.754 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-04-04T09:24:46.776 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-04-04T09:24:46.892 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-04-04T09:24:46.894 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-04-04T09:24:46.913 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-04-04T09:24:47.029 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-04-04T09:24:47.031 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-04-04T09:24:47.041 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-04-04T09:24:47.170 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-04-04T09:24:47.172 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-04-04T09:24:47.296 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-04-04T09:24:47.298 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-04-04T09:24:47.308 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-04-04T09:24:47.435 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-04-04T09:24:47.448 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-04-04T09:24:47.563 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-04-04T09:24:47.575 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-04-04T09:24:47.703 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-04-04T09:24:47.704 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-04-04T09:24:47.831 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-04-04T09:24:47.833 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-04-04T09:24:47.978 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-04-04T09:24:47.980 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-04-04T09:24:48.104 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-04-04T09:24:48.105 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-04-04T09:24:48.107 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-04-04T09:24:48.231 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-04-04T09:24:48.263 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-04-04T09:24:48.367 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-04-04T09:24:48.389 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-04-04T09:24:48.504 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-04-04T09:24:48.516 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-04-04T09:24:48.631 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-04-04T09:24:48.663 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-04-04T09:24:48.759 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-04-04T09:24:48.812 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-04-04T09:24:48.896 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-04-04T09:24:48.970 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-04-04T09:24:49.023 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-04-04T09:24:49.096 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-04-04T09:24:49.098 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-04-04T09:24:49.149 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-04-04T09:24:49.242 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-04-04T09:24:49.295 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-04-04T09:24:49.368 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-04-04T09:24:49.431 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-04-04T09:24:49.506 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-04-04T09:24:49.581 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-04-04T09:24:49.645 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-04-04T09:24:49.709 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-04-04T09:24:49.783 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-04-04T09:24:49.847 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-04-04T09:24:49.849 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-04-04T09:24:49.911 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-04-04T09:24:49.975 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-04-04T09:24:50.039 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-04-04T09:24:50.102 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-04-04T09:24:50.176 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-04-04T09:24:50.239 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-04-04T09:24:50.326 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-04-04T09:24:50.390 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-04-04T09:24:50.454 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-04-04T09:24:50.530 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-04-04T09:24:50.593 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-04-04T09:24:50.667 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-04-04T09:24:50.732 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-04-04T09:24:50.797 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-04-04T09:24:50.861 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-04-04T09:24:50.936 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-04-04T09:24:50.991 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-04-04T09:24:51.079 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-04-04T09:24:51.134 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-04-04T09:24:51.209 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-04-04T09:24:51.284 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-04-04T09:24:51.347 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-04-04T09:24:51.434 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-04-04T09:24:51.475 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-04-04T09:24:51.580 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-04-04T09:24:51.612 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-04-04T09:24:51.729 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-04-04T09:24:51.731 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-04-04T09:24:51.741 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-04-04T09:24:51.859 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-04-04T09:24:51.881 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-04-04T09:24:51.986 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-04-04T09:24:52.020 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-04-04T09:24:52.139 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-04-04T09:24:52.162 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-04-04T09:24:52.164 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-04-04T09:24:52.268 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-04-04T09:24:52.311 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-04-04T09:24:52.416 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-04-04T09:24:52.448 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-04-04T09:24:52.533 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:52 +0000 (0:00:17.609) 0:00:49.550 ********** 2021-04-04T09:24:52.582 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-04-04T09:24:52.625 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-04-04T09:24:52.627 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:52.637 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:52.650 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:52.652 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:52.662 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:52.663 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:52.674 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:52.676 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:52.686 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:52.707 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-04-04T09:24:52.709 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-04-04T09:24:52.825 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-04-04T09:24:52.827 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e 2021-04-04T09:24:52.828 INFO:teuthology.task.ansible.out:1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:52.837 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:52.870 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-04-04T09:24:52.966 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-04-04T09:24:52.968 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-04-04T09:24:52.989 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-04-04T09:24:52.991 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-04-04T09:24:53.107 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-04-04T09:24:53.109 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfad 2021-04-04T09:24:53.111 INFO:teuthology.task.ansible.out:a28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.119 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.131 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.142 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.144 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.154 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-04-04T09:24:53.156 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc 2021-04-04T09:24:53.157 INFO:teuthology.task.ansible.out:29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.166 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.177 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.179 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.189 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.191 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.200 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.201 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.214 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.216 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.225 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.236 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.238 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.248 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.250 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.261 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.273 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.275 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.285 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.287 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.297 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.299 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-04-04T09:24:53.300 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.314 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.324 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.326 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.337 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.352 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.353 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.365 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.366 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.377 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.379 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.389 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.391 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.402 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.409 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.417 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.428 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-04-04T09:24:53.430 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.441 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.443 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.452 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.463 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.464 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.475 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.487 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.488 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.499 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.510 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.521 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.523 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.533 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.544 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.546 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.556 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.557 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.569 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.580 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.582 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-04-04T09:24:53.582 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.584 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.594 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.596 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.606 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.606 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.617 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.619 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.629 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.631 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.644 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.656 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.657 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.668 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.680 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.682 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.691 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.693 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.706 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.708 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.718 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.720 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.730 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-04-04T09:24:53.732 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMm 2021-04-04T09:24:53.734 INFO:teuthology.task.ansible.out:F9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.743 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.745 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.755 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.766 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.772 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.777 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.778 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.790 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.792 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.803 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.814 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.825 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.827 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.837 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.848 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.859 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.861 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.872 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.873 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.884 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.895 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.897 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.907 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.919 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.921 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.931 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.933 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.944 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.956 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.956 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.968 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.980 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.981 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:53.993 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:54.049 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:54 +0000 (0:00:01.516) 0:00:51.066 ********** 2021-04-04T09:24:54.059 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-04-04T09:24:54.061 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-04-04T09:24:54.147 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-04-04T09:24:54.149 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi122.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMT 2021-04-04T09:24:54.151 INFO:teuthology.task.ansible.out:haZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-04-04T09:24:54.182 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:54 +0000 (0:00:00.133) 0:00:51.200 ********** 2021-04-04T09:24:54.285 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-04-04T09:24:54.413 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-04-04T09:24:54.424 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:24:54.426 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-04-04T09:24:54.455 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-04-04T09:24:54.509 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:54 +0000 (0:00:00.326) 0:00:51.527 ********** 2021-04-04T09:24:54.559 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:24:54.561 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-04-04T09:24:54.613 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-04-04T09:24:54.615 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:54.625 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:54.636 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:54.638 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:54.649 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:54.649 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:54.661 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:54.663 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:54.694 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:24:54.696 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-04-04T09:24:54.812 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-04-04T09:24:54.814 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e 2021-04-04T09:24:54.817 INFO:teuthology.task.ansible.out:1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:54.825 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:54.827 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi078.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-04-04T09:24:54.962 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-04-04T09:24:54.984 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-04-04T09:24:54.986 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-04-04T09:24:55.099 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:24:55.101 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-04-04T09:24:55.132 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-04-04T09:24:55.134 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfad 2021-04-04T09:24:55.135 INFO:teuthology.task.ansible.out:a28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.145 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.146 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.156 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.167 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.169 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.179 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.181 INFO:teuthology.task.ansible.out: skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.191 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.193 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.203 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.214 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.216 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.225 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.237 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.238 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} TASK [Remove revoked users] **************************************************** ok: [smithi078.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-04-04T09:24:55.239 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.250 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.261 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.262 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.272 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.274 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.284 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.285 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.295 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.306 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.308 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.317 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.320 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.329 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.331 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.341 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.343 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.352 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.353 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.364 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.375 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.377 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} TASK [Remove revoked users] **************************************************** ok: [smithi078.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-04-04T09:24:55.388 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-04-04T09:24:55.390 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.391 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.402 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.414 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.416 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.426 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.427 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.437 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.448 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.450 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.460 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.462 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.472 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.483 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.494 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.497 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.505 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.507 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.517 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi078.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-04-04T09:24:55.519 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.529 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.540 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.542 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.552 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.553 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.564 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.565 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.575 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.587 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.599 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.600 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.611 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.623 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.624 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.635 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.637 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.647 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.648 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.658 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:24:55.660 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.671 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.672 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.683 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.694 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.696 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.707 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.708 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.718 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.720 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.730 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.741 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.742 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.754 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.755 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.766 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.768 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.780 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.790 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.792 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} TASK [Remove revoked users] **************************************************** ok: [smithi078.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-04-04T09:24:55.803 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.805 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.814 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.825 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.827 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.837 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.848 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.850 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.859 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.871 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.882 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.884 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.893 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.904 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.906 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.916 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.927 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.928 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi078.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.939 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.941 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.951 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.961 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.963 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.973 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.984 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:55.995 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:56.006 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:56.047 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:56 +0000 (0:00:01.537) 0:00:53.065 ********** 2021-04-04T09:24:56.074 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:24:56.075 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-04-04T09:24:56.127 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-04-04T09:24:56.129 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:24:56.157 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:24:56 +0000 (0:00:00.110) 0:00:53.175 ********** 2021-04-04T09:24:56.206 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:24:56.208 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-04-04T09:24:56.344 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-04-04T09:24:56.377 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-04-04T09:24:56.485 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-04-04T09:24:56.517 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-04-04T09:24:56.613 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-04-04T09:24:56.656 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-04-04T09:24:56.753 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-04-04T09:24:56.816 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-04-04T09:24:56.891 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-04-04T09:24:56.988 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-04-04T09:24:57.032 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-04-04T09:24:57.148 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-04-04T09:24:57.171 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-04-04T09:24:57.287 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-04-04T09:24:57.309 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-04-04T09:24:57.446 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-04-04T09:24:57.458 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-04-04T09:24:57.595 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-04-04T09:24:57.596 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-04-04T09:24:57.735 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-04-04T09:24:57.746 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-04-04T09:24:57.883 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-04-04T09:24:57.885 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-04-04T09:24:58.021 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-04-04T09:24:58.032 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-04-04T09:24:58.159 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-04-04T09:24:58.214 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-04-04T09:24:58.299 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-04-04T09:24:58.355 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-04-04T09:24:58.440 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-04-04T09:24:58.514 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-04-04T09:24:58.588 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-04-04T09:24:58.674 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-04-04T09:24:58.727 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-04-04T09:24:58.835 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-04-04T09:24:58.867 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-04-04T09:24:58.994 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-04-04T09:24:59.006 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-04-04T09:24:59.143 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-04-04T09:24:59.156 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-04-04T09:24:59.294 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-04-04T09:24:59.296 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-04-04T09:24:59.423 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-04-04T09:24:59.445 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-04-04T09:24:59.585 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-04-04T09:24:59.597 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-04-04T09:24:59.725 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-04-04T09:24:59.874 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-04-04T09:25:00.023 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-04-04T09:25:00.161 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-04-04T09:25:00.288 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jdillaman", "name": "jdillaman", "remove": false, "state": "absent"} 2021-04-04T09:25:00.321 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-04-04T09:25:00.439 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-04-04T09:25:00.482 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-04-04T09:25:00.579 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-04-04T09:25:00.643 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-04-04T09:25:00.721 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-04-04T09:25:00.806 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-04-04T09:25:00.871 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-04-04T09:25:00.945 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-04-04T09:25:01.010 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-04-04T09:25:01.095 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-04-04T09:25:01.150 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-04-04T09:25:01.265 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-04-04T09:25:01.298 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-04-04T09:25:01.416 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-04-04T09:25:01.438 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-04-04T09:25:01.559 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-04-04T09:25:01.571 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-04-04T09:25:01.699 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-04-04T09:25:01.721 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-04-04T09:25:01.772 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:01 +0000 (0:00:05.615) 0:00:58.790 ********** 2021-04-04T09:25:01.853 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-04-04T09:25:02.012 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-04-04T09:25:02.195 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-04-04T09:25:02.197 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "checksum": "53b31205b3cc9729fea6ce5ceab6366ba2fdabc0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "6cd40c867036c949fadb741414d81285", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1617528301.8516634-19974-95978669969550/source", "state": "file", "uid": 0} 2021-04-04T09:25:02.226 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:02 +0000 (0:00:00.453) 0:00:59.244 ********** 2021-04-04T09:25:02.305 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-04-04T09:25:02.307 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => { "msg": "Host smithi078.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-04-04T09:25:02.332 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:02 +0000 (0:00:00.105) 0:00:59.350 ********** 2021-04-04T09:25:02.413 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-04-04T09:25:02.415 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-04-04T09:25:02.437 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:02 +0000 (0:00:00.105) 0:00:59.455 ********** 2021-04-04T09:25:02.495 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:25:02.497 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jdillaman", "name": "jdillaman", "remove": false, "state": "absent"} TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:02.520 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:02 +0000 (0:00:00.082) 0:00:59.538 ********** 2021-04-04T09:25:02.578 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-04-04T09:25:02.580 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:02.605 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:02 +0000 (0:00:00.085) 0:00:59.623 ********** 2021-04-04T09:25:02.647 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:25:02.649 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-04-04T09:25:02.691 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-04-04T09:25:02.693 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:02.719 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:02 +0000 (0:00:00.113) 0:00:59.737 ********** 2021-04-04T09:25:02.791 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-04-04T09:25:02.793 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:02.820 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:02 +0000 (0:00:00.100) 0:00:59.837 ********** 2021-04-04T09:25:02.830 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:25:02.831 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-04-04T09:25:02.896 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:02.922 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:02 +0000 (0:00:00.101) 0:00:59.939 ********** 2021-04-04T09:25:02.960 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:25:02.962 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-04-04T09:25:02.993 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-04-04T09:25:02.995 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:03.018 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:03 +0000 (0:00:00.096) 0:01:00.036 ********** 2021-04-04T09:25:03.099 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-04-04T09:25:03.101 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:03.123 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:03 +0000 (0:00:00.104) 0:01:00.140 ********** 2021-04-04T09:25:03.130 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:25:03.131 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-04-04T09:25:03.205 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:03.230 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:03 +0000 (0:00:00.107) 0:01:00.248 ********** 2021-04-04T09:25:03.248 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:25:03.250 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-04-04T09:25:03.308 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-04-04T09:25:03.310 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi078.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-04T09:25:03.338 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:03 +0000 (0:00:00.108) 0:01:00.356 ********** 2021-04-04T09:25:03.378 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:25:03.380 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-04-04T09:25:03.412 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:03.438 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:03 +0000 (0:00:00.099) 0:01:00.456 ********** 2021-04-04T09:25:03.510 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-04-04T09:25:03.512 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:03.534 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:03 +0000 (0:00:00.095) 0:01:00.552 ********** 2021-04-04T09:25:03.541 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:25:03.543 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-04-04T09:25:03.606 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-04-04T09:25:03.608 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-04T09:25:03.629 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:03 +0000 (0:00:00.095) 0:01:00.647 ********** 2021-04-04T09:25:03.689 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:25:03.691 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-04-04T09:25:03.701 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-04-04T09:25:03.703 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:03.730 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:03 +0000 (0:00:00.100) 0:01:00.747 ********** 2021-04-04T09:25:03.799 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-04-04T09:25:03.801 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-04T09:25:03.834 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:03 +0000 (0:00:00.104) 0:01:00.852 ********** 2021-04-04T09:25:03.852 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:25:03.853 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-04-04T09:25:03.917 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:03.943 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:03 +0000 (0:00:00.108) 0:01:00.960 ********** 2021-04-04T09:25:04.014 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-04-04T09:25:04.017 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-04-04T09:25:04.041 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:04.065 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:04 +0000 (0:00:00.122) 0:01:01.083 ********** 2021-04-04T09:25:04.091 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:04 +0000 (0:00:00.026) 0:01:01.109 ********** 2021-04-04T09:25:04.140 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-04-04T09:25:04.142 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:04.165 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:04 +0000 (0:00:00.073) 0:01:01.183 ********** 2021-04-04T09:25:04.235 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-04-04T09:25:04.237 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:04.265 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:04 +0000 (0:00:00.100) 0:01:01.283 ********** 2021-04-04T09:25:04.336 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi078.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-04T09:25:04.362 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:04 +0000 (0:00:00.096) 0:01:01.379 ********** 2021-04-04T09:25:04.420 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-04-04T09:25:04.422 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:04.446 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:04 +0000 (0:00:00.084) 0:01:01.464 ********** 2021-04-04T09:25:04.506 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-04-04T09:25:04.507 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "checksum": "53b31205b3cc9729fea6ce5ceab6366ba2fdabc0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "6cd40c867036c949fadb741414d81285", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1617528304.1732879-20231-105897519262311/source", "state": "file", "uid": 0} 2021-04-04T09:25:04.532 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:04 +0000 (0:00:00.085) 0:01:01.549 ********** 2021-04-04T09:25:04.539 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:04.563 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:04 +0000 (0:00:00.031) 0:01:01.581 ********** 2021-04-04T09:25:04.612 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-04-04T09:25:04.614 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => { "msg": "Host smithi122.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-04-04T09:25:04.638 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:04 +0000 (0:00:00.075) 0:01:01.656 ********** 2021-04-04T09:25:04.646 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-04-04T09:25:04.648 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:04.674 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:04 +0000 (0:00:00.035) 0:01:01.692 ********** 2021-04-04T09:25:04.711 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-04-04T09:25:04.713 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-04-04T09:25:04.736 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:04 +0000 (0:00:00.062) 0:01:01.754 ********** 2021-04-04T09:25:04.742 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-04-04T09:25:04.744 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:04.769 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:04 +0000 (0:00:00.032) 0:01:01.787 ********** 2021-04-04T09:25:04.808 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-04-04T09:25:04.809 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:04.843 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:04 +0000 (0:00:00.073) 0:01:01.860 ********** 2021-04-04T09:25:04.851 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-04-04T09:25:04.852 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:04.887 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:04 +0000 (0:00:00.044) 0:01:01.905 ********** 2021-04-04T09:25:04.915 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-04-04T09:25:04.917 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:04.943 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:04 +0000 (0:00:00.056) 0:01:01.961 ********** 2021-04-04T09:25:04.950 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-04-04T09:25:04.952 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:04.977 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:04 +0000 (0:00:00.034) 0:01:01.995 ********** 2021-04-04T09:25:05.026 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-04-04T09:25:05.028 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:05.060 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:05 +0000 (0:00:00.082) 0:01:02.078 ********** 2021-04-04T09:25:05.067 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-04-04T09:25:05.068 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-04T09:25:05.097 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:05 +0000 (0:00:00.036) 0:01:02.114 ********** 2021-04-04T09:25:05.135 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-04-04T09:25:05.137 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:05.163 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:05 +0000 (0:00:00.065) 0:01:02.180 ********** 2021-04-04T09:25:05.180 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-04-04T09:25:05.182 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:05.209 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:05 +0000 (0:00:00.046) 0:01:02.227 ********** 2021-04-04T09:25:05.239 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-04-04T09:25:05.241 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:05.263 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:05 +0000 (0:00:00.054) 0:01:02.281 ********** 2021-04-04T09:25:05.290 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-04-04T09:25:05.292 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:05.329 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:05 +0000 (0:00:00.065) 0:01:02.346 ********** 2021-04-04T09:25:05.348 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-04-04T09:25:05.350 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:05.384 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:05 +0000 (0:00:00.055) 0:01:02.402 ********** 2021-04-04T09:25:05.401 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-04-04T09:25:05.404 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:05.429 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:05 +0000 (0:00:00.045) 0:01:02.447 ********** 2021-04-04T09:25:05.472 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-04-04T09:25:05.474 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:05.497 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:05 +0000 (0:00:00.068) 0:01:02.515 ********** 2021-04-04T09:25:05.542 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-04-04T09:25:05.544 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:05.586 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:05 +0000 (0:00:00.088) 0:01:02.603 ********** 2021-04-04T09:25:05.592 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-04-04T09:25:05.594 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:05.618 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:05 +0000 (0:00:00.032) 0:01:02.636 ********** 2021-04-04T09:25:05.656 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-04-04T09:25:05.658 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:05.668 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:05.694 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:05 +0000 (0:00:00.076) 0:01:02.712 ********** 2021-04-04T09:25:05.710 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-04-04T09:25:05.712 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi122.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-04T09:25:05.737 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:05 +0000 (0:00:00.042) 0:01:02.754 ********** 2021-04-04T09:25:05.774 INFO:teuthology.task.ansible.out: 2021-04-04T09:25:05.776 INFO:teuthology.task.ansible.out:TASK [common : Clean yum cache] ************************************************ skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:05.809 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:05 +0000 (0:00:00.072) 0:01:02.827 ********** 2021-04-04T09:25:05.831 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:05.874 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:05 +0000 (0:00:00.065) 0:01:02.892 ********** 2021-04-04T09:25:05.977 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-04-04T09:25:05.979 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:06.003 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:05 +0000 (0:00:00.128) 0:01:03.021 ********** 2021-04-04T09:25:06.097 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.024233", "end": "2021-04-04 09:25:06.062190", "rc": 0, "start": "2021-04-04 09:25:06.037957", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-04-04T09:25:06.121 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:06 +0000 (0:00:00.117) 0:01:03.139 ********** 2021-04-04T09:25:06.128 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-04-04T09:25:06.129 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-04T09:25:06.155 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:06 +0000 (0:00:00.033) 0:01:03.172 ********** 2021-04-04T09:25:06.207 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-04-04T09:25:06.209 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:06.248 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:06 +0000 (0:00:00.092) 0:01:03.265 ********** 2021-04-04T09:25:06.258 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-04-04T09:25:06.259 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:06.296 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:06 +0000 (0:00:00.048) 0:01:03.313 ********** 2021-04-04T09:25:06.334 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-04-04T09:25:06.336 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:06.378 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:06 +0000 (0:00:00.082) 0:01:03.396 ********** 2021-04-04T09:25:06.410 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-04-04T09:25:06.411 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-04T09:25:06.439 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:06 +0000 (0:00:00.060) 0:01:03.456 ********** 2021-04-04T09:25:06.477 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-04-04T09:25:06.479 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:06.489 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:06.491 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:06.515 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:06 +0000 (0:00:00.076) 0:01:03.533 ********** 2021-04-04T09:25:06.533 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-04-04T09:25:06.535 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:06.559 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:06 +0000 (0:00:00.043) 0:01:03.576 ********** 2021-04-04T09:25:06.630 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-04-04T09:25:06.632 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:06.657 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:06 +0000 (0:00:00.097) 0:01:03.674 ********** 2021-04-04T09:25:06.663 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-04-04T09:25:06.665 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:06.689 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:06 +0000 (0:00:00.032) 0:01:03.707 ********** 2021-04-04T09:25:06.729 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-04-04T09:25:06.731 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:06.766 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:06 +0000 (0:00:00.076) 0:01:03.784 ********** 2021-04-04T09:25:06.783 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-04-04T09:25:06.785 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:06.808 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:06 +0000 (0:00:00.042) 0:01:03.826 ********** 2021-04-04T09:25:06.836 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-04-04T09:25:06.838 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:06.861 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:06 +0000 (0:00:00.053) 0:01:03.879 ********** 2021-04-04T09:25:06.890 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-04-04T09:25:06.892 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:06.915 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:06 +0000 (0:00:00.053) 0:01:03.933 ********** 2021-04-04T09:25:06.944 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-04-04T09:25:06.945 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:06.968 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:06 +0000 (0:00:00.052) 0:01:03.985 ********** 2021-04-04T09:25:06.985 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-04-04T09:25:06.987 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-04T09:25:07.012 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:07 +0000 (0:00:00.044) 0:01:04.030 ********** 2021-04-04T09:25:07.050 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-04-04T09:25:07.052 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:07.075 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:07 +0000 (0:00:00.063) 0:01:04.093 ********** 2021-04-04T09:25:07.094 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-04-04T09:25:07.097 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:07.132 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:07 +0000 (0:00:00.050) 0:01:04.143 ********** 2021-04-04T09:25:07.195 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-04-04T09:25:07.197 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:07.220 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:07 +0000 (0:00:00.094) 0:01:04.238 ********** 2021-04-04T09:25:07.302 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-04-04T09:25:07.304 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:07.340 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:07 +0000 (0:00:00.119) 0:01:04.357 ********** 2021-04-04T09:25:07.410 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:07.433 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:07 +0000 (0:00:00.093) 0:01:04.451 ********** 2021-04-04T09:25:07.513 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-04-04T09:25:07.514 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:07.537 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:07 +0000 (0:00:00.104) 0:01:04.555 ********** 2021-04-04T09:25:07.545 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-04-04T09:25:07.546 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.304441", "end": "2021-04-04 09:25:07.500479", "rc": 0, "start": "2021-04-04 09:25:07.196038", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:25:07.577 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:07 +0000 (0:00:00.039) 0:01:04.594 ********** 2021-04-04T09:25:07.626 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-04-04T09:25:07.628 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:07.660 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:07 +0000 (0:00:00.083) 0:01:04.678 ********** 2021-04-04T09:25:07.667 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-04-04T09:25:07.669 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:07.693 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:07 +0000 (0:00:00.033) 0:01:04.711 ********** 2021-04-04T09:25:07.732 INFO:teuthology.task.ansible.out: 2021-04-04T09:25:07.733 INFO:teuthology.task.ansible.out:TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi122.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-04T09:25:07.758 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:07 +0000 (0:00:00.064) 0:01:04.776 ********** 2021-04-04T09:25:07.839 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-04-04T09:25:07.841 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:07.867 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:07 +0000 (0:00:00.108) 0:01:04.885 ********** 2021-04-04T09:25:07.940 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-04-04T09:25:07.942 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:07.965 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:07 +0000 (0:00:00.098) 0:01:04.983 ********** 2021-04-04T09:25:07.972 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-04-04T09:25:07.974 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.060797", "end": "2021-04-04 09:25:07.921434", "rc": 0, "start": "2021-04-04 09:25:07.860637", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:25:07.999 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:07 +0000 (0:00:00.033) 0:01:05.016 ********** 2021-04-04T09:25:08.079 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-04-04T09:25:08.081 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:08.110 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:08 +0000 (0:00:00.111) 0:01:05.128 ********** 2021-04-04T09:25:08.192 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-04-04T09:25:08.193 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:08.218 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:08 +0000 (0:00:00.107) 0:01:05.236 ********** 2021-04-04T09:25:08.288 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-04-04T09:25:08.289 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:08.299 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:08.323 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:08 +0000 (0:00:00.105) 0:01:05.341 ********** 2021-04-04T09:25:08.396 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-04-04T09:25:08.397 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:08.421 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:08 +0000 (0:00:00.097) 0:01:05.439 ********** 2021-04-04T09:25:08.630 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.013899", "end": "2021-04-04 09:25:08.981147", "rc": 0, "start": "2021-04-04 09:25:08.967248", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-04-04T09:25:08.654 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:08 +0000 (0:00:00.232) 0:01:05.671 ********** 2021-04-04T09:25:08.722 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-04-04T09:25:08.723 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:08.746 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:08 +0000 (0:00:00.091) 0:01:05.763 ********** 2021-04-04T09:25:08.836 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-04-04T09:25:08.838 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:08.861 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:08 +0000 (0:00:00.115) 0:01:05.879 ********** 2021-04-04T09:25:08.943 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-04-04T09:25:08.945 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:08.956 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:08.958 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:08.982 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:08 +0000 (0:00:00.120) 0:01:05.999 ********** 2021-04-04T09:25:09.051 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-04-04T09:25:09.053 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:09.075 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:09 +0000 (0:00:00.093) 0:01:06.093 ********** 2021-04-04T09:25:09.155 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-04-04T09:25:09.157 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:09.179 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:09 +0000 (0:00:00.103) 0:01:06.196 ********** 2021-04-04T09:25:09.248 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-04-04T09:25:09.250 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:09.272 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:09 +0000 (0:00:00.093) 0:01:06.290 ********** 2021-04-04T09:25:09.362 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-04-04T09:25:09.365 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:09.386 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:09 +0000 (0:00:00.113) 0:01:06.403 ********** 2021-04-04T09:25:09.475 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-04-04T09:25:09.477 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:09.502 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:09 +0000 (0:00:00.116) 0:01:06.519 ********** 2021-04-04T09:25:10.149 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-04-04T09:25:10.150 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.509420", "end": "2021-04-04 09:25:10.500422", "rc": 0, "start": "2021-04-04 09:25:09.991002", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:25:10.172 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:10 +0000 (0:00:00.670) 0:01:07.190 ********** 2021-04-04T09:25:10.241 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-04-04T09:25:10.243 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:10.266 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:10 +0000 (0:00:00.094) 0:01:07.284 ********** 2021-04-04T09:25:10.492 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-04-04T09:25:10.494 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.036659", "end": "2021-04-04 09:25:10.840841", "rc": 0, "start": "2021-04-04 09:25:10.804182", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:25:10.516 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:10 +0000 (0:00:00.249) 0:01:07.534 ********** 2021-04-04T09:25:12.150 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-04-04T09:25:12.152 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1617528026, "cache_updated": false, "changed": false} 2021-04-04T09:25:12.181 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:12 +0000 (0:00:01.664) 0:01:09.198 ********** 2021-04-04T09:25:13.128 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-04-04T09:25:13.130 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1617528026, "cache_updated": false, "changed": false} 2021-04-04T09:25:13.155 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:13 +0000 (0:00:00.974) 0:01:10.172 ********** 2021-04-04T09:25:13.224 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-04-04T09:25:13.226 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:13.260 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:13 +0000 (0:00:00.105) 0:01:10.278 ********** 2021-04-04T09:25:13.969 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-04-04T09:25:13.971 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-04-04T09:25:13.993 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:13 +0000 (0:00:00.732) 0:01:11.011 ********** 2021-04-04T09:25:14.115 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-04-04T09:25:14.117 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1617528070, "cache_updated": false, "changed": false} 2021-04-04T09:25:14.149 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:14 +0000 (0:00:00.156) 0:01:11.167 ********** 2021-04-04T09:25:15.053 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-04-04T09:25:15.055 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1617528026, "cache_updated": false, "changed": false} 2021-04-04T09:25:15.077 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:15 +0000 (0:00:00.927) 0:01:12.095 ********** 2021-04-04T09:25:15.369 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-04-04T09:25:15.371 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"cache_update_time": 1617528070, "cache_updated": false, "changed": false} 2021-04-04T09:25:15.399 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:15 +0000 (0:00:00.322) 0:01:12.417 ********** 2021-04-04T09:25:15.470 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-04-04T09:25:15.472 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:15.497 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:15 +0000 (0:00:00.098) 0:01:12.515 ********** 2021-04-04T09:25:15.504 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-04-04T09:25:15.507 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-04-04T09:25:15.833 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-04-04T09:25:15.865 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-04-04T09:25:15.867 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-04-04T09:25:15.890 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:15 +0000 (0:00:00.392) 0:01:12.907 ********** 2021-04-04T09:25:16.159 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-04-04T09:25:16.161 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-04-04T09:25:16.186 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:16 +0000 (0:00:00.296) 0:01:13.204 ********** 2021-04-04T09:25:16.320 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-04-04T09:25:16.573 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} Sunday 04 April 2021 09:25:16 +0000 (0:00:00.163) 0:01:13.368 ********** 2021-04-04T09:25:16.876 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-04-04T09:25:16.878 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"cache_update_time": 1617528070, "cache_updated": false, "changed": false} 2021-04-04T09:25:16.911 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:16 +0000 (0:00:00.560) 0:01:13.928 ********** 2021-04-04T09:25:16.921 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-04-04T09:25:16.922 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-04-04T09:25:17.200 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-04-04T09:25:17.473 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-04-04T09:25:17.499 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:17 +0000 (0:00:00.587) 0:01:14.516 ********** 2021-04-04T09:25:17.516 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-04-04T09:25:17.517 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-04-04T09:25:17.707 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-04-04T09:25:17.709 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-04-04T09:25:17.733 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:17 +0000 (0:00:00.234) 0:01:14.751 ********** 2021-04-04T09:25:17.833 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-04-04T09:25:17.835 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-04-04T09:25:17.907 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-04-04T09:25:17.909 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:18.029 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:18 +0000 (0:00:00.295) 0:01:15.046 ********** 2021-04-04T09:25:18.099 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-04-04T09:25:18.100 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-04T09:25:18.125 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:18 +0000 (0:00:00.096) 0:01:15.142 ********** 2021-04-04T09:25:18.142 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-04-04T09:25:18.144 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-04-04T09:25:18.171 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:18 +0000 (0:00:00.046) 0:01:15.188 ********** 2021-04-04T09:25:18.199 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-04-04T09:25:18.201 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:18.211 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:18.213 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:18.222 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:18.250 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:18 +0000 (0:00:00.079) 0:01:15.268 ********** 2021-04-04T09:25:18.310 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-04-04T09:25:18.311 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-04-04T09:25:18.334 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:18 +0000 (0:00:00.084) 0:01:15.352 ********** 2021-04-04T09:25:18.340 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-04-04T09:25:18.342 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:18.368 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:18 +0000 (0:00:00.033) 0:01:15.386 ********** 2021-04-04T09:25:18.714 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-04-04T09:25:18.716 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-04-04T09:25:18.989 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-04-04T09:25:19.281 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-04-04T09:25:19.306 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:19 +0000 (0:00:00.938) 0:01:16.324 ********** 2021-04-04T09:25:19.322 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-04-04T09:25:19.324 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1617528026, "cache_updated": false, "changed": false} 2021-04-04T09:25:19.349 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:19 +0000 (0:00:00.042) 0:01:16.367 ********** 2021-04-04T09:25:19.388 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-04-04T09:25:19.389 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-04-04T09:25:19.417 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:19 +0000 (0:00:00.067) 0:01:16.434 ********** 2021-04-04T09:25:19.486 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-04-04T09:25:19.488 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:19.509 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:19 +0000 (0:00:00.092) 0:01:16.527 ********** 2021-04-04T09:25:19.578 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-04-04T09:25:19.579 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-04T09:25:19.603 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:19 +0000 (0:00:00.093) 0:01:16.621 ********** 2021-04-04T09:25:19.694 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-04-04T09:25:19.695 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:19.706 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:19.708 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:19.730 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:19 +0000 (0:00:00.127) 0:01:16.748 ********** 2021-04-04T09:25:19.736 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-04-04T09:25:19.738 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-04-04T09:25:19.762 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:19 +0000 (0:00:00.031) 0:01:16.780 ********** 2021-04-04T09:25:19.799 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-04-04T09:25:19.801 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:19.827 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:19 +0000 (0:00:00.065) 0:01:16.845 ********** 2021-04-04T09:25:20.116 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-04-04T09:25:20.117 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-04-04T09:25:20.142 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:20 +0000 (0:00:00.314) 0:01:17.160 ********** 2021-04-04T09:25:20.337 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-04-04T09:25:20.338 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-04-04T09:25:20.453 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-04-04T09:25:20.477 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:20 +0000 (0:00:00.334) 0:01:17.495 ********** 2021-04-04T09:25:20.555 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-04-04T09:25:20.557 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:20.578 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:20 +0000 (0:00:00.101) 0:01:17.596 ********** 2021-04-04T09:25:20.646 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-04-04T09:25:20.648 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-04T09:25:20.670 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:20 +0000 (0:00:00.091) 0:01:17.688 ********** 2021-04-04T09:25:20.738 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-04-04T09:25:20.740 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:20.768 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:20 +0000 (0:00:00.097) 0:01:17.786 ********** 2021-04-04T09:25:20.858 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-04-04T09:25:20.860 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"cache_update_time": 1617528070, "cache_updated": false, "changed": false} 2021-04-04T09:25:20.884 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:20 +0000 (0:00:00.115) 0:01:17.902 ********** 2021-04-04T09:25:21.131 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-04-04T09:25:21.132 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-04-04T09:25:21.159 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:21 +0000 (0:00:00.274) 0:01:18.176 ********** 2021-04-04T09:25:21.269 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-04-04T09:25:21.271 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-04-04T09:25:21.294 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:21 +0000 (0:00:00.135) 0:01:18.312 ********** 2021-04-04T09:25:21.737 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-04-04T09:25:21.739 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-04-04T09:25:21.760 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:21 +0000 (0:00:00.465) 0:01:18.777 ********** 2021-04-04T09:25:21.851 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-04-04T09:25:21.852 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2021-04-04 09:23:35 UTC", "ActiveEnterTimestampMonotonic": "19957157", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service nss-lookup.target remote-fs.target time-sync.target local-fs.target basic.target system.slice -.mount var-run.mount network.target sysinit.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2021-04-04 09:23:35 UTC", "AssertTimestampMonotonic": "19956447", "Before": "multi-user.target xdm.service shutdown.target plymouth-quit.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2021-04-04 09:23:35 UTC", "ConditionTimestampMonotonic": "19956447", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-04-04T09:25:21.854 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1468", "ExecMainStartTimestamp": "Sun 2021-04-04 09:23:35 UTC", "ExecMainStartTimestampMonotonic": "19957133", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Sun 2021-04-04 09:23:35 UTC] ; stop_time=[n/a] ; pid=1468 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2021-04-04 09:23:35 UTC", "InactiveExitTimestampMonotonic": "19957157", "InvocationID": "3d5f0454518d4751861c4304a0f2ffb3", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2021-04-04T09:25:21.855 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128028", "LimitNPROCSoft": "128028", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128028", "LimitSIGPENDINGSoft": "128028", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1468", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2021-04-04T09:25:21.857 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2021-04-04 09:23:35 UTC", "StateChangeTimestampMonotonic": "19957157", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2021-04-04 09:23:35 UTC", "WatchdogTimestampMonotonic": "19957156", "WatchdogUSec": "0"}} 2021-04-04T09:25:21.882 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:21 +0000 (0:00:00.122) 0:01:18.900 ********** 2021-04-04T09:25:21.884 INFO:teuthology.task.ansible.out: 2021-04-04T09:25:21.953 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-04-04T09:25:21.955 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-04-04T09:25:21.956 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:21.981 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:21 +0000 (0:00:00.098) 0:01:18.998 ********** 2021-04-04T09:25:22.048 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-04-04T09:25:22.050 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:22.075 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:22 +0000 (0:00:00.094) 0:01:19.092 ********** 2021-04-04T09:25:22.081 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-04-04T09:25:22.083 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-04-04T09:25:22.111 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:22 +0000 (0:00:00.036) 0:01:19.129 ********** 2021-04-04T09:25:22.170 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-04-04T09:25:22.172 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:22.197 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:22 +0000 (0:00:00.085) 0:01:19.214 ********** 2021-04-04T09:25:22.203 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-04-04T09:25:22.205 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:22.229 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:22 +0000 (0:00:00.032) 0:01:19.247 ********** 2021-04-04T09:25:22.265 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-04-04T09:25:22.267 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:22.291 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:22 +0000 (0:00:00.061) 0:01:19.309 ********** 2021-04-04T09:25:22.307 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-04-04T09:25:22.309 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-04-04T09:25:22.333 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:22 +0000 (0:00:00.042) 0:01:19.351 ********** 2021-04-04T09:25:22.360 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-04-04T09:25:22.362 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:22.385 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:22 +0000 (0:00:00.051) 0:01:19.403 ********** 2021-04-04T09:25:22.411 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-04-04T09:25:22.413 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:22.436 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:22 +0000 (0:00:00.051) 0:01:19.454 ********** 2021-04-04T09:25:22.452 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-04-04T09:25:22.454 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:22.478 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:22 +0000 (0:00:00.042) 0:01:19.496 ********** 2021-04-04T09:25:22.547 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-04-04T09:25:22.548 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:22.573 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:22 +0000 (0:00:00.094) 0:01:19.591 ********** 2021-04-04T09:25:22.641 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-04-04T09:25:22.642 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:22.668 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:22 +0000 (0:00:00.095) 0:01:19.686 ********** 2021-04-04T09:25:22.737 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-04-04T09:25:22.739 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:22.765 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:22 +0000 (0:00:00.096) 0:01:19.782 ********** 2021-04-04T09:25:22.782 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-04-04T09:25:22.783 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-04-04T09:25:22.808 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:22 +0000 (0:00:00.043) 0:01:19.826 ********** 2021-04-04T09:25:22.836 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-04-04T09:25:22.837 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:22.866 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:22 +0000 (0:00:00.057) 0:01:19.883 ********** 2021-04-04T09:25:22.946 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-04-04T09:25:22.948 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-04-04T09:25:22.973 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:22 +0000 (0:00:00.107) 0:01:19.991 ********** 2021-04-04T09:25:23.052 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-04-04T09:25:23.054 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-04-04T09:25:23.077 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:23 +0000 (0:00:00.104) 0:01:20.095 ********** 2021-04-04T09:25:23.085 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-04-04T09:25:23.086 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2021-04-04 09:23:45 UTC", "ActiveEnterTimestampMonotonic": "19822979", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "local-fs.target remote-fs.target basic.target -.mount network.target nss-lookup.target var-run.mount systemd-journald.socket systemd-tmpfiles-setup.service sysinit.target time-sync.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2021-04-04 09:23:45 UTC", "AssertTimestampMonotonic": "19822330", "Before": "plymouth-quit.service shutdown.target getty@tty1.service xdm.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2021-04-04 09:23:45 UTC", "ConditionTimestampMonotonic": "19822328", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-04-04T09:25:23.088 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1340", "ExecMainStartTimestamp": "Sun 2021-04-04 09:23:45 UTC", "ExecMainStartTimestampMonotonic": "19822954", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Sun 2021-04-04 09:23:45 UTC] ; stop_time=[n/a] ; pid=1340 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2021-04-04 09:23:45 UTC", "InactiveExitTimestampMonotonic": "19822979", "InvocationID": "a8bdd4aae2b44bd3837972ae9f614177", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2021-04-04T09:25:23.090 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128028", "LimitNPROCSoft": "128028", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128028", "LimitSIGPENDINGSoft": "128028", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1340", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2021-04-04T09:25:23.091 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2021-04-04 09:23:45 UTC", "StateChangeTimestampMonotonic": "19822979", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2021-04-04 09:23:45 UTC", "WatchdogTimestampMonotonic": "19822977", "WatchdogUSec": "0"}} 2021-04-04T09:25:23.113 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:23 +0000 (0:00:00.035) 0:01:20.131 ********** 2021-04-04T09:25:23.182 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-04-04T09:25:23.183 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Including major version specific variables.] ****************** ok: [smithi078.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-04-04T09:25:23.207 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:23 +0000 (0:00:00.093) 0:01:20.224 ********** 2021-04-04T09:25:23.228 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:23 +0000 (0:00:00.021) 0:01:20.246 ********** 2021-04-04T09:25:23.298 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-04-04T09:25:23.300 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-04-04T09:25:23.323 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:23 +0000 (0:00:00.095) 0:01:20.341 ********** 2021-04-04T09:25:23.330 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-04-04T09:25:23.331 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:23.362 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:23 +0000 (0:00:00.038) 0:01:20.380 ********** 2021-04-04T09:25:23.430 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-04-04T09:25:23.432 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:23.457 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:23 +0000 (0:00:00.094) 0:01:20.475 ********** 2021-04-04T09:25:23.464 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-04-04T09:25:23.466 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-04-04T09:25:23.495 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:23 +0000 (0:00:00.038) 0:01:20.513 ********** 2021-04-04T09:25:23.523 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-04-04T09:25:23.524 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:23.553 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:23 +0000 (0:00:00.057) 0:01:20.571 ********** 2021-04-04T09:25:23.626 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-04-04T09:25:23.628 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:23.660 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:23 +0000 (0:00:00.106) 0:01:20.677 ********** 2021-04-04T09:25:23.689 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-04-04T09:25:23.691 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-04-04T09:25:23.720 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:23 +0000 (0:00:00.060) 0:01:20.737 ********** 2021-04-04T09:25:23.750 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-04-04T09:25:23.752 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:23.776 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:23 +0000 (0:00:00.056) 0:01:20.794 ********** 2021-04-04T09:25:23.803 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-04-04T09:25:23.805 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:23.828 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:23 +0000 (0:00:00.051) 0:01:20.846 ********** 2021-04-04T09:25:23.855 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-04-04T09:25:23.857 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:23.881 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:23 +0000 (0:00:00.052) 0:01:20.899 ********** 2021-04-04T09:25:23.919 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-04-04T09:25:23.921 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:23.945 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:23 +0000 (0:00:00.064) 0:01:20.963 ********** 2021-04-04T09:25:23.952 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-04-04T09:25:23.954 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:23.980 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:23 +0000 (0:00:00.034) 0:01:20.998 ********** 2021-04-04T09:25:24.049 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-04-04T09:25:24.051 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:24.074 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:24 +0000 (0:00:00.094) 0:01:21.092 ********** 2021-04-04T09:25:24.154 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-04-04T09:25:24.155 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-04-04T09:25:24.178 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:24 +0000 (0:00:00.103) 0:01:21.196 ********** 2021-04-04T09:25:24.185 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-04-04T09:25:24.186 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:24.214 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:24 +0000 (0:00:00.035) 0:01:21.232 ********** 2021-04-04T09:25:24.293 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-04-04T09:25:24.295 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-04-04T09:25:24.317 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:24 +0000 (0:00:00.103) 0:01:21.335 ********** 2021-04-04T09:25:24.417 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-04-04T09:25:24.419 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-04-04T09:25:24.445 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:24 +0000 (0:00:00.127) 0:01:21.463 ********** 2021-04-04T09:25:24.524 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-04-04T09:25:24.525 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-04-04T09:25:24.548 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:24 +0000 (0:00:00.103) 0:01:21.566 ********** 2021-04-04T09:25:24.628 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-04-04T09:25:24.630 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-04-04T09:25:24.658 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:24 +0000 (0:00:00.109) 0:01:21.676 ********** 2021-04-04T09:25:24.679 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:24 +0000 (0:00:00.021) 0:01:21.697 ********** 2021-04-04T09:25:24.729 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-04-04T09:25:24.730 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi078"}, "changed": false} 2021-04-04T09:25:24.754 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:24 +0000 (0:00:00.074) 0:01:21.772 ********** 2021-04-04T09:25:24.815 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-04-04T09:25:24.817 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-04-04T09:25:24.842 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:24 +0000 (0:00:00.088) 0:01:21.860 ********** 2021-04-04T09:25:25.040 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-04-04T09:25:25.042 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-04-04T09:25:25.064 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:25 +0000 (0:00:00.221) 0:01:22.082 ********** 2021-04-04T09:25:25.144 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-04-04T09:25:25.146 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:25.169 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:25 +0000 (0:00:00.104) 0:01:22.186 ********** 2021-04-04T09:25:25.269 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-04-04T09:25:25.271 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:25.298 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:25 +0000 (0:00:00.129) 0:01:22.316 ********** 2021-04-04T09:25:25.514 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-04-04T09:25:25.516 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-04-04T09:25:25.538 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:25 +0000 (0:00:00.239) 0:01:22.556 ********** 2021-04-04T09:25:25.899 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-04-04T09:25:25.901 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi078.front.sepia.ceph.com", "ansible_hostname": "smithi078", "ansible_nodename": "smithi078"}, "changed": true, "name": "smithi078"} 2021-04-04T09:25:25.926 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:25 +0000 (0:00:00.388) 0:01:22.944 ********** 2021-04-04T09:25:26.045 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-04-04T09:25:26.047 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-04-04T09:25:26.072 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:26 +0000 (0:00:00.145) 0:01:23.089 ********** 2021-04-04T09:25:26.165 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-04-04T09:25:26.167 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi122"}, "changed": false} 2021-04-04T09:25:26.197 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:26 +0000 (0:00:00.125) 0:01:23.214 ********** 2021-04-04T09:25:26.320 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-04-04T09:25:26.322 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-04-04T09:25:26.346 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:26 +0000 (0:00:00.149) 0:01:23.364 ********** 2021-04-04T09:25:26.734 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-04-04T09:25:26.735 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-04-04T09:25:26.766 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:26 +0000 (0:00:00.419) 0:01:23.784 ********** 2021-04-04T09:25:26.774 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-04-04T09:25:26.776 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi122.front.sepia.ceph.com", "ansible_hostname": "smithi122", "ansible_nodename": "smithi122"}, "changed": true, "name": "smithi122"} 2021-04-04T09:25:26.813 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:26 +0000 (0:00:00.047) 0:01:23.831 ********** 2021-04-04T09:25:27.260 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-04-04T09:25:27.262 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-04-04T09:25:27.284 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:27 +0000 (0:00:00.470) 0:01:24.302 ********** 2021-04-04T09:25:27.325 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-04-04T09:25:27.326 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-04-04T09:25:27.353 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:27 +0000 (0:00:00.068) 0:01:24.371 ********** 2021-04-04T09:25:27.424 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-04-04T09:25:27.426 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:27.461 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:27 +0000 (0:00:00.108) 0:01:24.479 ********** 2021-04-04T09:25:27.533 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-04-04T09:25:27.534 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:27.559 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:27 +0000 (0:00:00.097) 0:01:24.577 ********** 2021-04-04T09:25:27.641 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-04-04T09:25:27.643 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:27.680 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:27 +0000 (0:00:00.120) 0:01:24.697 ********** 2021-04-04T09:25:27.688 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-04-04T09:25:27.717 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:27 +0000 (0:00:00.037) 0:01:24.735 ********** 2021-04-04T09:25:27.755 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-04-04T09:25:27.757 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:27.786 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:27 +0000 (0:00:00.068) 0:01:24.803 ********** 2021-04-04T09:25:27.856 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-04-04T09:25:27.859 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:27.881 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:27 +0000 (0:00:00.094) 0:01:24.898 ********** 2021-04-04T09:25:27.962 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-04-04T09:25:27.964 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:27.992 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:27 +0000 (0:00:00.111) 0:01:25.010 ********** 2021-04-04T09:25:28.072 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-04-04T09:25:28.074 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:28.097 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:28 +0000 (0:00:00.104) 0:01:25.114 ********** 2021-04-04T09:25:28.167 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-04-04T09:25:28.168 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:28.201 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:28 +0000 (0:00:00.104) 0:01:25.219 ********** 2021-04-04T09:25:28.273 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-04-04T09:25:28.275 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-04-04T09:25:28.303 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:28 +0000 (0:00:00.101) 0:01:25.321 ********** 2021-04-04T09:25:28.311 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-04-04T09:25:28.343 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:28 +0000 (0:00:00.040) 0:01:25.361 ********** 2021-04-04T09:25:28.374 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-04-04T09:25:28.376 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:28.401 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:28 +0000 (0:00:00.058) 0:01:25.419 ********** 2021-04-04T09:25:28.418 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-04-04T09:25:28.447 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:28 +0000 (0:00:00.045) 0:01:25.465 ********** 2021-04-04T09:25:28.476 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-04-04T09:25:28.478 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:28.501 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:28 +0000 (0:00:00.053) 0:01:25.519 ********** 2021-04-04T09:25:28.519 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-04-04T09:25:28.543 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:28 +0000 (0:00:00.041) 0:01:25.561 ********** 2021-04-04T09:25:28.581 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-04-04T09:25:28.583 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:28.607 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:28 +0000 (0:00:00.063) 0:01:25.625 ********** 2021-04-04T09:25:28.624 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-04-04T09:25:28.626 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:28.653 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:28 +0000 (0:00:00.046) 0:01:25.671 ********** 2021-04-04T09:25:28.691 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-04-04T09:25:28.693 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:28.721 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:28 +0000 (0:00:00.067) 0:01:25.739 ********** 2021-04-04T09:25:28.740 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-04-04T09:25:28.742 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:28.769 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:28 +0000 (0:00:00.048) 0:01:25.787 ********** 2021-04-04T09:25:28.797 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-04-04T09:25:28.799 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:28.822 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:28 +0000 (0:00:00.052) 0:01:25.839 ********** 2021-04-04T09:25:28.839 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-04-04T09:25:28.841 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:28.876 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:28 +0000 (0:00:00.054) 0:01:25.894 ********** 2021-04-04T09:25:28.895 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-04-04T09:25:28.897 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:28.923 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:28 +0000 (0:00:00.046) 0:01:25.940 ********** 2021-04-04T09:25:28.973 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-04T09:25:28.974 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.001 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:28 +0000 (0:00:00.077) 0:01:26.018 ********** 2021-04-04T09:25:29.007 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-04-04T09:25:29.009 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.033 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.032) 0:01:26.051 ********** 2021-04-04T09:25:29.082 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-04-04T09:25:29.084 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.112 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.079) 0:01:26.130 ********** 2021-04-04T09:25:29.119 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-04-04T09:25:29.121 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.145 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.032) 0:01:26.163 ********** 2021-04-04T09:25:29.203 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-04-04T09:25:29.205 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.228 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.082) 0:01:26.246 ********** 2021-04-04T09:25:29.237 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-04-04T09:25:29.264 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.035) 0:01:26.281 ********** 2021-04-04T09:25:29.302 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-04-04T09:25:29.304 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.334 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.069) 0:01:26.351 ********** 2021-04-04T09:25:29.340 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-04-04T09:25:29.372 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.038) 0:01:26.390 ********** 2021-04-04T09:25:29.401 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-04-04T09:25:29.403 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.427 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.054) 0:01:26.445 ********** 2021-04-04T09:25:29.444 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-04-04T09:25:29.473 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.045) 0:01:26.490 ********** 2021-04-04T09:25:29.500 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-04-04T09:25:29.502 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.526 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.053) 0:01:26.544 ********** 2021-04-04T09:25:29.543 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-04-04T09:25:29.545 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.571 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.045) 0:01:26.589 ********** 2021-04-04T09:25:29.598 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-04-04T09:25:29.600 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.625 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.053) 0:01:26.642 ********** 2021-04-04T09:25:29.663 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-04-04T09:25:29.664 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.674 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.705 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.080) 0:01:26.723 ********** 2021-04-04T09:25:29.711 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-04-04T09:25:29.713 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.737 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.032) 0:01:26.755 ********** 2021-04-04T09:25:29.777 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-04-04T09:25:29.779 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.802 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.064) 0:01:26.820 ********** 2021-04-04T09:25:29.818 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-04-04T09:25:29.820 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.848 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.045) 0:01:26.865 ********** 2021-04-04T09:25:29.866 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-04T09:25:29.868 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.893 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.045) 0:01:26.910 ********** 2021-04-04T09:25:29.920 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-04-04T09:25:29.922 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.949 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.056) 0:01:26.967 ********** 2021-04-04T09:25:29.966 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-04-04T09:25:29.968 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:29.994 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:29 +0000 (0:00:00.045) 0:01:27.012 ********** 2021-04-04T09:25:30.012 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-04-04T09:25:30.014 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.039 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.045) 0:01:27.057 ********** 2021-04-04T09:25:30.089 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-04-04T09:25:30.090 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.115 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.075) 0:01:27.132 ********** 2021-04-04T09:25:30.121 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-04-04T09:25:30.122 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.157 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.042) 0:01:27.175 ********** 2021-04-04T09:25:30.177 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-04-04T09:25:30.178 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.203 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.045) 0:01:27.220 ********** 2021-04-04T09:25:30.230 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-04-04T09:25:30.232 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.267 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.064) 0:01:27.285 ********** 2021-04-04T09:25:30.286 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-04-04T09:25:30.287 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.313 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.045) 0:01:27.330 ********** 2021-04-04T09:25:30.350 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-04-04T09:25:30.352 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.375 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.062) 0:01:27.393 ********** 2021-04-04T09:25:30.392 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-04-04T09:25:30.394 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.420 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.044) 0:01:27.438 ********** 2021-04-04T09:25:30.471 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-04-04T09:25:30.472 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.497 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.076) 0:01:27.515 ********** 2021-04-04T09:25:30.504 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-04-04T09:25:30.506 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.530 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.032) 0:01:27.548 ********** 2021-04-04T09:25:30.570 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-04-04T09:25:30.572 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.598 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.067) 0:01:27.616 ********** 2021-04-04T09:25:30.615 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-04-04T09:25:30.616 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.646 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.047) 0:01:27.664 ********** 2021-04-04T09:25:30.663 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-04-04T09:25:30.665 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.695 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.049) 0:01:27.713 ********** 2021-04-04T09:25:30.725 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-04-04T09:25:30.727 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.755 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.059) 0:01:27.772 ********** 2021-04-04T09:25:30.772 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-04-04T09:25:30.774 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.803 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.047) 0:01:27.820 ********** 2021-04-04T09:25:30.832 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-04-04T09:25:30.834 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.860 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.056) 0:01:27.877 ********** 2021-04-04T09:25:30.879 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-04-04T09:25:30.881 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.904 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.044) 0:01:27.922 ********** 2021-04-04T09:25:30.942 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-04-04T09:25:30.944 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:30.967 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:30 +0000 (0:00:00.063) 0:01:27.985 ********** 2021-04-04T09:25:31.052 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-04-04T09:25:31.054 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:31.076 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:31 +0000 (0:00:00.108) 0:01:28.094 ********** 2021-04-04T09:25:31.103 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-04-04T09:25:31.105 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.004868", "end": "2021-04-04 09:25:31.085900", "rc": 0, "start": "2021-04-04 09:25:31.081032", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:25:31.129 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:31 +0000 (0:00:00.052) 0:01:28.147 ********** 2021-04-04T09:25:31.148 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-04-04T09:25:31.150 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:31.174 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:31 +0000 (0:00:00.044) 0:01:28.192 ********** 2021-04-04T09:25:31.201 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-04-04T09:25:31.235 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:31 +0000 (0:00:00.061) 0:01:28.253 ********** 2021-04-04T09:25:31.244 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-04-04T09:25:31.246 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:31.276 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:31 +0000 (0:00:00.041) 0:01:28.294 ********** 2021-04-04T09:25:31.349 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-04-04T09:25:31.351 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:31.374 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:31 +0000 (0:00:00.097) 0:01:28.391 ********** 2021-04-04T09:25:31.442 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-04-04T09:25:31.444 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:31.467 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:31 +0000 (0:00:00.093) 0:01:28.485 ********** 2021-04-04T09:25:31.547 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-04-04T09:25:31.549 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:31.572 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:31 +0000 (0:00:00.104) 0:01:28.590 ********** 2021-04-04T09:25:31.621 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-04-04T09:25:31.623 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-04-04T09:25:31.647 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:31 +0000 (0:00:00.074) 0:01:28.665 ********** 2021-04-04T09:25:31.654 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-04-04T09:25:31.655 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:31.682 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:31 +0000 (0:00:00.033) 0:01:28.698 ********** 2021-04-04T09:25:31.741 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-04-04T09:25:31.743 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:31.766 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:31 +0000 (0:00:00.086) 0:01:28.784 ********** 2021-04-04T09:25:31.773 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-04-04T09:25:31.775 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:31.804 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:31 +0000 (0:00:00.037) 0:01:28.822 ********** 2021-04-04T09:25:32.001 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-04-04T09:25:32.002 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.004836", "end": "2021-04-04 09:25:32.343950", "rc": 0, "start": "2021-04-04 09:25:32.339114", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:25:32.025 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:32 +0000 (0:00:00.220) 0:01:29.042 ********** 2021-04-04T09:25:32.104 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-04-04T09:25:32.132 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:32 +0000 (0:00:00.107) 0:01:29.150 ********** 2021-04-04T09:25:32.517 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-04-04T09:25:32.518 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-04-04T09:25:32.546 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:32 +0000 (0:00:00.414) 0:01:29.564 ********** 2021-04-04T09:25:32.617 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-04-04T09:25:32.619 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:32.644 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:32 +0000 (0:00:00.097) 0:01:29.662 ********** 2021-04-04T09:25:33.127 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-04-04T09:25:33.129 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-04-04T09:25:33.843 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-04-04T09:25:34.065 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-04-04T09:25:34.092 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:34 +0000 (0:00:01.447) 0:01:31.109 ********** 2021-04-04T09:25:34.737 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-04-04T09:25:34.762 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:34 +0000 (0:00:00.670) 0:01:31.780 ********** 2021-04-04T09:25:35.084 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-04-04T09:25:35.086 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1617528026, "cache_updated": false, "changed": false} 2021-04-04T09:25:35.116 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:35 +0000 (0:00:00.352) 0:01:32.132 ********** 2021-04-04T09:25:35.195 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-04-04T09:25:35.221 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:35 +0000 (0:00:00.106) 0:01:32.239 ********** 2021-04-04T09:25:35.448 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-04-04T09:25:35.450 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.037927", "end": "2021-04-04 09:25:35.421471", "rc": 0, "start": "2021-04-04 09:25:35.383544", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:25:35.473 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:35 +0000 (0:00:00.252) 0:01:32.491 ********** 2021-04-04T09:25:35.824 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-04-04T09:25:35.826 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"cache_update_time": 1617528070, "cache_updated": false, "changed": false} 2021-04-04T09:25:35.849 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:35 +0000 (0:00:00.375) 0:01:32.866 ********** 2021-04-04T09:25:35.919 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-04-04T09:25:35.945 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:35 +0000 (0:00:00.096) 0:01:32.963 ********** 2021-04-04T09:25:36.184 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-04-04T09:25:36.186 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.037166", "end": "2021-04-04 09:25:36.520700", "rc": 0, "start": "2021-04-04 09:25:36.483534", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:25:36.209 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:36 +0000 (0:00:00.262) 0:01:33.226 ********** 2021-04-04T09:25:38.453 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-04-04T09:25:38.455 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1617528026, "cache_updated": false, "changed": false} 2021-04-04T09:25:38.489 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:38 +0000 (0:00:02.280) 0:01:35.507 ********** 2021-04-04T09:25:39.341 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"cache_update_time": 1617528070, "cache_updated": false, "changed": false} 2021-04-04T09:25:39.374 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:39 +0000 (0:00:00.884) 0:01:36.392 ********** 2021-04-04T09:25:48.692 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-04-04T09:25:48.693 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 36 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137595 files and directorie 2021-04-04T09:25:48.695 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 36 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-04-04T09:25:48.697 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137595 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-04-04T09:25:48.722 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:48 +0000 (0:00:09.347) 0:01:45.739 ********** 2021-04-04T09:25:49.004 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 36 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137595 files and directorie 2021-04-04T09:25:49.006 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 36 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-04-04T09:25:49.008 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137595 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-04-04T09:25:49.041 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:49 +0000 (0:00:00.319) 0:01:46.058 ********** 2021-04-04T09:25:49.678 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-04-04T09:25:49.680 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1617528026, "cache_updated": false, "changed": false} 2021-04-04T09:25:49.703 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:49 +0000 (0:00:00.662) 0:01:46.721 ********** 2021-04-04T09:25:49.773 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-04-04T09:25:49.775 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:49.796 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:49 +0000 (0:00:00.092) 0:01:46.814 ********** 2021-04-04T09:25:50.022 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-04-04T09:25:50.024 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"cache_update_time": 1617528070, "cache_updated": false, "changed": false} 2021-04-04T09:25:50.049 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:50 +0000 (0:00:00.252) 0:01:47.067 ********** 2021-04-04T09:25:50.130 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-04-04T09:25:50.132 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:25:50.156 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:25:50 +0000 (0:00:00.107) 0:01:47.174 ********** 2021-04-04T09:26:00.276 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-04-04T09:26:00.277 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1617528026, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk\n gitweb git-cvs git-mediawiki git-svn samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\nThe following packages will be upgraded:\n git\n1 upgraded, 6 newly installed, 0 to remove and 35 not upgraded.\nNeed to get 14.3 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.8 [3916 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]\nGet:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]\nFetched 14.3 MB in 1s (11.7 MB/s)\nSelecting previously unselected package librados2.\r\n(Reading database .. 2021-04-04T09:26:00.278 INFO:teuthology.task.ansible.out:. \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137453 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nPreparing to unpack .../4-git_1%3a2.17.1-1ubuntu0.8_amd64.deb ...\r\nUnpacking git (1:2.17.1-1ubuntu0.8) over (1:2.17.1-1ubuntu0.7) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../5-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../6-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up git (1:2.17.1-1ubuntu0.8) ...\r\nSetti 2021-04-04T09:26:00.280 INFO:teuthology.task.ansible.out:ng up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk", " gitweb git-cvs git-mediawiki git-svn samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "The following packages will be upgraded:", " git", "1 upgraded, 6 newly installed, 0 to remove and 35 not upgraded.", "Need to get 14.3 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.8 [3916 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]", "Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.1 2021-04-04T09:26:00.281 INFO:teuthology.task.ansible.out:1+dfsg-1ubuntu7.36 [870 kB]", "Fetched 14.3 MB in 1s (11.7 MB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137453 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Preparing to unpack .../4-git_1%3a2.17.1-1ubuntu0.8_amd64.deb ...", "Unpacking git (1:2.17.1-1ubuntu0.8) over (1:2.17.1-1ubuntu0.7) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../5-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../6-qemu-utils_1%3a2.11+dfsg-1ub 2021-04-04T09:26:00.283 INFO:teuthology.task.ansible.out:untu7.36_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up git (1:2.17.1-1ubuntu0.8) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-04-04T09:26:00.304 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:00 +0000 (0:00:10.148) 0:01:57.322 ********** 2021-04-04T09:26:00.513 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"cache_update_time": 1617528070, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk\n gitweb git-cvs git-mediawiki git-svn samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\nThe following packages will be upgraded:\n git\n1 upgraded, 6 newly installed, 0 to remove and 35 not upgraded.\nNeed to get 14.3 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.8 [3916 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]\nGet:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]\nFetched 14.3 MB in 1s (12.0 MB/s)\nSelecting previously unselected package librados2.\r\n(Reading database .. 2021-04-04T09:26:00.515 INFO:teuthology.task.ansible.out:. \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137453 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nPreparing to unpack .../4-git_1%3a2.17.1-1ubuntu0.8_amd64.deb ...\r\nUnpacking git (1:2.17.1-1ubuntu0.8) over (1:2.17.1-1ubuntu0.7) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../5-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../6-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up git (1:2.17.1-1ubuntu0.8) ...\r\nSetti 2021-04-04T09:26:00.517 INFO:teuthology.task.ansible.out:ng up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk", " gitweb git-cvs git-mediawiki git-svn samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "The following packages will be upgraded:", " git", "1 upgraded, 6 newly installed, 0 to remove and 35 not upgraded.", "Need to get 14.3 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.8 [3916 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]", "Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.1 2021-04-04T09:26:00.518 INFO:teuthology.task.ansible.out:1+dfsg-1ubuntu7.36 [870 kB]", "Fetched 14.3 MB in 1s (12.0 MB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137453 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Preparing to unpack .../4-git_1%3a2.17.1-1ubuntu0.8_amd64.deb ...", "Unpacking git (1:2.17.1-1ubuntu0.8) over (1:2.17.1-1ubuntu0.7) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../5-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../6-qemu-utils_1%3a2.11+dfsg-1ub 2021-04-04T09:26:00.520 INFO:teuthology.task.ansible.out:untu7.36_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up git (1:2.17.1-1ubuntu0.8) ...", "Setting up li 2021-04-04T09:26:00.520 INFO:teuthology.task.ansible.out:brbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-04-04T09:26:00.546 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:00 +0000 (0:00:00.241) 0:01:57.563 ********** 2021-04-04T09:26:01.253 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-04-04T09:26:01.255 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1617528026, "cache_updated": false, "changed": false} 2021-04-04T09:26:01.278 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:01 +0000 (0:00:00.732) 0:01:58.296 ********** 2021-04-04T09:26:01.483 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"cache_update_time": 1617528070, "cache_updated": false, "changed": false} 2021-04-04T09:26:01.507 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:01 +0000 (0:00:00.228) 0:01:58.525 ********** 2021-04-04T09:26:02.196 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-04-04T09:26:02.198 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1617528026, "cache_updated": false, "changed": false} 2021-04-04T09:26:02.221 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:02 +0000 (0:00:00.713) 0:01:59.239 ********** 2021-04-04T09:26:02.404 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-04-04T09:26:02.406 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-04-04T09:26:02.428 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:02 +0000 (0:00:00.207) 0:01:59.446 ********** 2021-04-04T09:26:02.444 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-04-04T09:26:02.447 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"cache_update_time": 1617528070, "cache_updated": false, "changed": false} 2021-04-04T09:26:02.469 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:02 +0000 (0:00:00.040) 0:01:59.487 ********** 2021-04-04T09:26:02.517 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-04-04T09:26:02.519 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:02.540 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:02 +0000 (0:00:00.071) 0:01:59.558 ********** 2021-04-04T09:26:02.673 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-04-04T09:26:02.674 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-04-04T09:26:02.699 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:02 +0000 (0:00:00.158) 0:01:59.716 ********** 2021-04-04T09:26:02.736 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-04-04T09:26:02.738 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-04-04T09:26:02.761 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:02 +0000 (0:00:00.062) 0:01:59.779 ********** 2021-04-04T09:26:02.789 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-04-04T09:26:02.790 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:02.818 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:02 +0000 (0:00:00.055) 0:01:59.835 ********** 2021-04-04T09:26:03.001 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-04-04T09:26:03.003 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-04-04T09:26:03.026 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:03 +0000 (0:00:00.208) 0:02:00.043 ********** 2021-04-04T09:26:03.157 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-04-04T09:26:03.159 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-04-04T09:26:03.196 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:03 +0000 (0:00:00.170) 0:02:00.214 ********** 2021-04-04T09:26:03.404 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-04-04T09:26:03.405 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-04-04T09:26:03.428 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:03 +0000 (0:00:00.232) 0:02:00.446 ********** 2021-04-04T09:26:03.435 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-04-04T09:26:03.437 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-04-04T09:26:03.464 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:03 +0000 (0:00:00.035) 0:02:00.481 ********** 2021-04-04T09:26:03.608 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-04-04T09:26:03.610 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-04T09:26:03.637 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:03 +0000 (0:00:00.173) 0:02:00.655 ********** 2021-04-04T09:26:03.675 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-04-04T09:26:03.677 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-04-04T09:26:03.702 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:03 +0000 (0:00:00.065) 0:02:00.720 ********** 2021-04-04T09:26:03.888 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-04-04T09:26:03.889 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-04T09:26:03.912 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:03 +0000 (0:00:00.210) 0:02:00.930 ********** 2021-04-04T09:26:03.963 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-04-04T09:26:03.965 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Sun 2021-04-04 09:23:36 UTC", "ActiveEnterTimestampMonotonic": "20455908", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target systemd-tmpfiles-setup.service remote-fs.target basic.target nss-lookup.target network.target system.slice systemd-journald.socket -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2021-04-04 09:23:35 UTC", "AssertTimestampMonotonic": "19958465", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2021-04-04 09:23:35 UTC", "ConditionTimestampMonotonic": "19958465", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-04-04T09:26:03.967 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1551", "ExecMainStartTimestamp": "Sun 2021-04-04 09:23:36 UTC", "ExecMainStartTimestampMonotonic": "20455897", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2021-04-04 09:23:35 UTC", "InactiveExitTimestampMonotonic": "19959131", "InvocationID": "eaa2f43d2bc8451a9db9a3d7d80033f4", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-04-04T09:26:03.969 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128028", "LimitNPROCSoft": "128028", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128028", "LimitSIGPENDINGSoft": "128028", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1551", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-04-04T09:26:03.971 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2021-04-04 09:23:36 UTC", "StateChangeTimestampMonotonic": "20455908", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2021-04-04 09:23:36 UTC", "WatchdogTimestampMonotonic": "20455907", "WatchdogUSec": "0"}} 2021-04-04T09:26:03.993 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:03 +0000 (0:00:00.080) 0:02:01.011 ********** 2021-04-04T09:26:04.074 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-04-04T09:26:04.075 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:04.101 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:04 +0000 (0:00:00.107) 0:02:01.118 ********** 2021-04-04T09:26:04.180 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-04-04T09:26:04.182 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:04.204 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:04 +0000 (0:00:00.103) 0:02:01.222 ********** 2021-04-04T09:26:04.264 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-04-04T09:26:04.265 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Sun 2021-04-04 09:23:45 UTC", "ActiveEnterTimestampMonotonic": "20220262", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service sysinit.target systemd-journald.socket remote-fs.target basic.target system.slice -.mount nss-lookup.target network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2021-04-04 09:23:45 UTC", "AssertTimestampMonotonic": "19821637", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2021-04-04 09:23:45 UTC", "ConditionTimestampMonotonic": "19821637", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-04-04T09:26:04.267 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1409", "ExecMainStartTimestamp": "Sun 2021-04-04 09:23:45 UTC", "ExecMainStartTimestampMonotonic": "20220252", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2021-04-04 09:23:45 UTC", "InactiveExitTimestampMonotonic": "19822287", "InvocationID": "a52baad8500347a5ae21e809a90d90d6", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-04-04T09:26:04.269 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128028", "LimitNPROCSoft": "128028", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128028", "LimitSIGPENDINGSoft": "128028", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1409", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-04-04T09:26:04.271 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2021-04-04 09:23:45 UTC", "StateChangeTimestampMonotonic": "20220262", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2021-04-04 09:23:45 UTC", "WatchdogTimestampMonotonic": "20220261", "WatchdogUSec": "0"}} 2021-04-04T09:26:04.290 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:04 +0000 (0:00:00.086) 0:02:01.308 ********** 2021-04-04T09:26:04.296 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-04-04T09:26:04.298 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:04.329 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:04 +0000 (0:00:00.038) 0:02:01.346 ********** 2021-04-04T09:26:04.379 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-04-04T09:26:04.381 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:04.410 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:04 +0000 (0:00:00.081) 0:02:01.428 ********** 2021-04-04T09:26:04.417 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-04-04T09:26:04.418 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:04.447 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:04 +0000 (0:00:00.036) 0:02:01.464 ********** 2021-04-04T09:26:04.500 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:04.537 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:04 +0000 (0:00:00.089) 0:02:01.554 ********** 2021-04-04T09:26:04.544 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-04-04T09:26:04.546 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:04.583 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:04 +0000 (0:00:00.045) 0:02:01.600 ********** 2021-04-04T09:26:04.613 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-04-04T09:26:04.615 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:04.640 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:04 +0000 (0:00:00.057) 0:02:01.658 ********** 2021-04-04T09:26:04.669 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-04-04T09:26:04.670 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:04.695 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:04 +0000 (0:00:00.054) 0:02:01.713 ********** 2021-04-04T09:26:04.725 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-04-04T09:26:04.727 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:04.761 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:04 +0000 (0:00:00.066) 0:02:01.779 ********** 2021-04-04T09:26:04.781 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-04-04T09:26:04.787 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:04.817 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:04 +0000 (0:00:00.055) 0:02:01.835 ********** 2021-04-04T09:26:04.846 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-04-04T09:26:04.848 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:04.873 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:04 +0000 (0:00:00.055) 0:02:01.890 ********** 2021-04-04T09:26:04.911 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-04-04T09:26:04.913 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:04.950 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:04 +0000 (0:00:00.077) 0:02:01.968 ********** 2021-04-04T09:26:04.967 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-04-04T09:26:04.970 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.006 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.056) 0:02:02.024 ********** 2021-04-04T09:26:05.025 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.067 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.059) 0:02:02.083 ********** 2021-04-04T09:26:05.088 INFO:teuthology.task.ansible.out: 2021-04-04T09:26:05.090 INFO:teuthology.task.ansible.out:TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.125 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.059) 0:02:02.143 ********** 2021-04-04T09:26:05.154 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-04-04T09:26:05.157 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.181 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.055) 0:02:02.199 ********** 2021-04-04T09:26:05.199 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-04-04T09:26:05.201 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.228 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.046) 0:02:02.246 ********** 2021-04-04T09:26:05.255 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-04-04T09:26:05.257 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.283 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.054) 0:02:02.301 ********** 2021-04-04T09:26:05.323 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-04T09:26:05.325 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.348 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.065) 0:02:02.366 ********** 2021-04-04T09:26:05.375 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-04-04T09:26:05.377 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.402 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.053) 0:02:02.419 ********** 2021-04-04T09:26:05.430 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-04-04T09:26:05.432 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.467 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.065) 0:02:02.485 ********** 2021-04-04T09:26:05.487 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-04-04T09:26:05.489 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.515 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.047) 0:02:02.532 ********** 2021-04-04T09:26:05.544 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-04-04T09:26:05.546 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.570 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.054) 0:02:02.587 ********** 2021-04-04T09:26:05.587 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-04-04T09:26:05.589 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.622 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.051) 0:02:02.639 ********** 2021-04-04T09:26:05.641 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-04-04T09:26:05.643 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.670 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.048) 0:02:02.687 ********** 2021-04-04T09:26:05.708 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-04-04T09:26:05.709 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.743 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.073) 0:02:02.761 ********** 2021-04-04T09:26:05.749 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-04-04T09:26:05.751 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.779 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.035) 0:02:02.797 ********** 2021-04-04T09:26:05.809 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-04-04T09:26:05.811 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.846 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.066) 0:02:02.864 ********** 2021-04-04T09:26:05.865 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-04-04T09:26:05.867 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.891 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.044) 0:02:02.909 ********** 2021-04-04T09:26:05.918 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-04-04T09:26:05.920 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:05.946 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:05 +0000 (0:00:00.054) 0:02:02.963 ********** 2021-04-04T09:26:05.974 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-04-04T09:26:05.976 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.010 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.064) 0:02:03.028 ********** 2021-04-04T09:26:06.028 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-04-04T09:26:06.030 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.056 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.045) 0:02:03.073 ********** 2021-04-04T09:26:06.095 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-04-04T09:26:06.097 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.122 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.065) 0:02:03.139 ********** 2021-04-04T09:26:06.129 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-04-04T09:26:06.130 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.155 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.033) 0:02:03.173 ********** 2021-04-04T09:26:06.204 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-04-04T09:26:06.205 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.233 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.078) 0:02:03.251 ********** 2021-04-04T09:26:06.261 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-04-04T09:26:06.263 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.287 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.053) 0:02:03.305 ********** 2021-04-04T09:26:06.304 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-04-04T09:26:06.306 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.333 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.045) 0:02:03.350 ********** 2021-04-04T09:26:06.364 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-04-04T09:26:06.366 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.394 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.061) 0:02:03.412 ********** 2021-04-04T09:26:06.411 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-04-04T09:26:06.412 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.440 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.046) 0:02:03.458 ********** 2021-04-04T09:26:06.468 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-04-04T09:26:06.470 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.495 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.054) 0:02:03.513 ********** 2021-04-04T09:26:06.512 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-04-04T09:26:06.514 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.538 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.043) 0:02:03.556 ********** 2021-04-04T09:26:06.565 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-04-04T09:26:06.567 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.591 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.052) 0:02:03.608 ********** 2021-04-04T09:26:06.609 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.637 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.046) 0:02:03.655 ********** 2021-04-04T09:26:06.666 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-04-04T09:26:06.668 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.702 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.064) 0:02:03.719 ********** 2021-04-04T09:26:06.722 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-04-04T09:26:06.724 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.760 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.058) 0:02:03.777 ********** 2021-04-04T09:26:06.777 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-04-04T09:26:06.779 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.804 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.044) 0:02:03.822 ********** 2021-04-04T09:26:06.863 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-04-04T09:26:06.865 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.890 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.085) 0:02:03.907 ********** 2021-04-04T09:26:06.897 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-04-04T09:26:06.899 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:06.924 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.034) 0:02:03.942 ********** 2021-04-04T09:26:06.973 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-04-04T09:26:06.974 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:07.000 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:06 +0000 (0:00:00.075) 0:02:04.017 ********** 2021-04-04T09:26:07.081 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:07.112 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:07 +0000 (0:00:00.112) 0:02:04.129 ********** 2021-04-04T09:26:07.181 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-04-04T09:26:07.183 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:07.209 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:07 +0000 (0:00:00.096) 0:02:04.226 ********** 2021-04-04T09:26:07.383 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-04-04T09:26:07.385 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-04-04T09:26:07.420 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:07 +0000 (0:00:00.211) 0:02:04.438 ********** 2021-04-04T09:26:07.429 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-04-04T09:26:07.462 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:07 +0000 (0:00:00.041) 0:02:04.479 ********** 2021-04-04T09:26:07.785 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-04-04T09:26:07.787 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-04-04T09:26:07.809 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:07 +0000 (0:00:00.347) 0:02:04.826 ********** 2021-04-04T09:26:07.847 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-04-04T09:26:07.872 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:07 +0000 (0:00:00.063) 0:02:04.890 ********** 2021-04-04T09:26:07.888 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-04-04T09:26:07.890 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:07.922 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:07 +0000 (0:00:00.049) 0:02:04.939 ********** 2021-04-04T09:26:07.982 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:08.020 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:08 +0000 (0:00:00.097) 0:02:05.037 ********** 2021-04-04T09:26:08.331 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-04-04T09:26:08.333 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-04-04T09:26:08.355 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:08 +0000 (0:00:00.336) 0:02:05.373 ********** 2021-04-04T09:26:08.414 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-04-04T09:26:08.444 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:08 +0000 (0:00:00.087) 0:02:05.461 ********** 2021-04-04T09:26:08.754 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-04-04T09:26:08.756 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-04-04T09:26:08.786 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:08 +0000 (0:00:00.342) 0:02:05.803 ********** 2021-04-04T09:26:08.814 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-04-04T09:26:08.841 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:08 +0000 (0:00:00.055) 0:02:05.858 ********** 2021-04-04T09:26:09.161 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-04-04T09:26:09.163 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-04-04T09:26:09.187 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:09 +0000 (0:00:00.346) 0:02:06.205 ********** 2021-04-04T09:26:09.215 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-04-04T09:26:09.243 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:09 +0000 (0:00:00.055) 0:02:06.261 ********** 2021-04-04T09:26:09.399 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** ok: [smithi122.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-04T09:26:09.425 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:09 +0000 (0:00:00.182) 0:02:06.443 ********** 2021-04-04T09:26:09.433 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-04T09:26:09.464 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:09 +0000 (0:00:00.038) 0:02:06.481 ********** 2021-04-04T09:26:09.492 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-04-04T09:26:09.493 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:09.529 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:09 +0000 (0:00:00.065) 0:02:06.546 ********** 2021-04-04T09:26:09.549 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:09.581 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:09 +0000 (0:00:00.052) 0:02:06.599 ********** 2021-04-04T09:26:09.609 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-04-04T09:26:09.611 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:09.636 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:09 +0000 (0:00:00.054) 0:02:06.654 ********** 2021-04-04T09:26:09.663 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:09.691 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:09 +0000 (0:00:00.054) 0:02:06.708 ********** 2021-04-04T09:26:09.707 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-04-04T09:26:09.709 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:09.735 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:09 +0000 (0:00:00.044) 0:02:06.753 ********** 2021-04-04T09:26:09.762 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:09.789 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:09 +0000 (0:00:00.053) 0:02:06.806 ********** 2021-04-04T09:26:09.805 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-04-04T09:26:09.807 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:09.831 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:09 +0000 (0:00:00.042) 0:02:06.849 ********** 2021-04-04T09:26:09.850 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:09.875 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:09 +0000 (0:00:00.043) 0:02:06.893 ********** 2021-04-04T09:26:09.892 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-04-04T09:26:09.894 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:09.918 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:09 +0000 (0:00:00.042) 0:02:06.936 ********** 2021-04-04T09:26:09.935 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:09.961 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:09 +0000 (0:00:00.042) 0:02:06.978 ********** 2021-04-04T09:26:09.988 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-04T09:26:09.989 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-04-04T09:26:10.017 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:10 +0000 (0:00:00.056) 0:02:07.035 ********** 2021-04-04T09:26:10.047 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-04-04T09:26:10.084 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:10 +0000 (0:00:00.066) 0:02:07.102 ********** 2021-04-04T09:26:10.114 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-04-04T09:26:10.116 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.127 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.128 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.139 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.140 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.152 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-04-04T09:26:10.154 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.164 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.166 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.177 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.179 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.179 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.190 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.192 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.203 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.205 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.206 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-04-04T09:26:10.247 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:10 +0000 (0:00:00.162) 0:02:07.265 ********** 2021-04-04T09:26:10.256 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.258 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.258 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.259 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.260 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.260 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.272 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.300 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:10 +0000 (0:00:00.053) 0:02:07.318 ********** 2021-04-04T09:26:10.508 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-04-04T09:26:10.510 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.002958", "end": "2021-04-04 09:26:10.482981", "rc": 0, "start": "2021-04-04 09:26:10.480023", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology-lock"]} 2021-04-04T09:26:10.550 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:10 +0000 (0:00:00.249) 0:02:07.568 ********** 2021-04-04T09:26:10.559 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.002881", "end": "2021-04-04 09:26:10.522306", "rc": 0, "start": "2021-04-04 09:26:10.519425", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology-lock"]} 2021-04-04T09:26:10.589 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:10 +0000 (0:00:00.038) 0:02:07.607 ********** 2021-04-04T09:26:10.637 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-04-04T09:26:10.639 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.681 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:10 +0000 (0:00:00.092) 0:02:07.699 ********** 2021-04-04T09:26:10.699 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.728 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:10 +0000 (0:00:00.047) 0:02:07.746 ********** 2021-04-04T09:26:10.758 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-04-04T09:26:10.759 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.797 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:10 +0000 (0:00:00.068) 0:02:07.814 ********** 2021-04-04T09:26:10.806 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.833 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:10 +0000 (0:00:00.036) 0:02:07.851 ********** 2021-04-04T09:26:10.881 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-04-04T09:26:10.884 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.910 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:10 +0000 (0:00:00.076) 0:02:07.928 ********** 2021-04-04T09:26:10.917 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:10.943 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:10 +0000 (0:00:00.032) 0:02:07.960 ********** 2021-04-04T09:26:11.851 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-04-04T09:26:11.853 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"cache_update_time": 1617528070, "cache_updated": false, "changed": false} 2021-04-04T09:26:11.874 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:11 +0000 (0:00:00.931) 0:02:08.892 ********** 2021-04-04T09:26:12.038 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1617528026, "cache_updated": false, "changed": false} 2021-04-04T09:26:12.063 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:12 +0000 (0:00:00.188) 0:02:09.080 ********** 2021-04-04T09:26:12.069 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-04-04T09:26:12.071 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:12.100 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:12 +0000 (0:00:00.036) 0:02:09.117 ********** 2021-04-04T09:26:12.201 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-04-04T09:26:12.203 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item={'block_used': 13760082, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226335026, 'size_available': 927068266496, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149513, 'block_size': 4096, 'inode_available': 60905463}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226335026, "block_size": 4096, "block_total": 240095108, "block_used": 13760082, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905463, "inode_total": 61054976, "inode_used": 149513, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927068266496, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}} 2021-04-04T09:26:12.229 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:12 +0000 (0:00:00.129) 0:02:09.247 ********** 2021-04-04T09:26:12.267 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-04-04T09:26:12.269 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:12.295 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:12 +0000 (0:00:00.065) 0:02:09.312 ********** 2021-04-04T09:26:12.396 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-04-04T09:26:12.398 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item={'block_used': 13760082, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226335026, 'size_available': 927068266496, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149513, 'block_size': 4096, 'inode_available': 60905463}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226335026, "block_size": 4096, "block_total": 240095108, "block_used": 13760082, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905463, "inode_total": 61054976, "inode_used": 149513, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927068266496, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}} 2021-04-04T09:26:12.423 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:12 +0000 (0:00:00.127) 0:02:09.440 ********** 2021-04-04T09:26:12.429 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-04-04T09:26:12.431 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.010071", "end": "2021-04-04 09:26:12.732279", "rc": 0, "start": "2021-04-04 09:26:12.722208", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:12.467 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:12 +0000 (0:00:00.044) 0:02:09.484 ********** 2021-04-04T09:26:12.558 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-04-04T09:26:12.560 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'block_used': 13760082, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226335026, 'size_available': 927068266496, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149513, 'block_size': 4096, 'inode_available': 60905463}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226335026, "block_size": 4096, "block_total": 240095108, "block_used": 13760082, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905463, "inode_total": 61054976, "inode_used": 149513, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927068266496, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:12.584 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:12 +0000 (0:00:00.117) 0:02:09.601 ********** 2021-04-04T09:26:12.631 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-04-04T09:26:12.633 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009812", "end": "2021-04-04 09:26:12.611002", "rc": 0, "start": "2021-04-04 09:26:12.601190", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:12.656 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:12 +0000 (0:00:00.072) 0:02:09.674 ********** 2021-04-04T09:26:12.662 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-04-04T09:26:12.689 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:12 +0000 (0:00:00.033) 0:02:09.707 ********** 2021-04-04T09:26:12.875 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-04-04T09:26:12.877 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'block_used': 13760082, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226335026, 'size_available': 927068266496, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149513, 'block_size': 4096, 'inode_available': 60905463}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226335026, "block_size": 4096, "block_total": 240095108, "block_used": 13760082, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905463, "inode_total": 61054976, "inode_used": 149513, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927068266496, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:12.900 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:12 +0000 (0:00:00.210) 0:02:09.917 ********** 2021-04-04T09:26:12.906 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-04T09:26:12.908 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:13.086 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-04-04T09:26:13.114 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:13 +0000 (0:00:00.214) 0:02:10.131 ********** 2021-04-04T09:26:13.323 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-04T09:26:13.325 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:14.754 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r93rOKGkskRR3v3Ws28vJRKw6vgarC3k6y'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.706349", "end": "2021-04-04 09:26:15.064612", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r93rOKGkskRR3v3Ws28vJRKw6vgarC3k6y"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:13.358263", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-04T09:26:14.767 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:14.778 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:14.946 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAloLUMJh7GPEpABmNY4zJrf7hFhfEUKzMHb'], 'uuids': ['b08d2f36-660e-43b9-acbe-256e7495825d']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.493248", "end": "2021-04-04 09:26:14.918097", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAloLUMJh7GPEpABmNY4zJrf7hFhfEUKzMHb"], "labels": [], "masters": [], "uuids": ["b08d2f36-660e-43b9-acbe-256e7495825d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:13.424849", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-04T09:26:14.959 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:14.971 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:16.024 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r9qgWll2yTcDOFPlBeycdKGESUvRDUVf0P'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.118282", "end": "2021-04-04 09:26:16.337572", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r9qgWll2yTcDOFPlBeycdKGESUvRDUVf0P"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:15.219290", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-04T09:26:16.036 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5L5J2', 'wwn-0x5000c500918e77a4'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5L5J2-part1', 'wwn-0x5000c500918e77a4-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918e77a4', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5L5J2', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L5J2", "wwn-0x5000c500918e77a4"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L5J2-part1", "wwn-0x5000c500918e77a4-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5L5J2", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918e77a4"}}, "skip_reason": 2021-04-04T09:26:16.038 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-04-04T09:26:16.203 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAlo93PXdF3Fj9MfDPLRM8HZLDA1xl5HcFX6'], 'uuids': ['dc850323-c9d1-430f-8004-e8d25d6e1dde']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.117057", "end": "2021-04-04 09:26:16.181626", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAlo93PXdF3Fj9MfDPLRM8HZLDA1xl5HcFX6"], "labels": [], "masters": [], "uuids": ["dc850323-c9d1-430f-8004-e8d25d6e1dde"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:15.064569", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-04T09:26:16.214 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LGR8', 'wwn-0x5000c500918e79d4'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LGR8-part1', 'wwn-0x5000c500918e79d4-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918e79d4', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LGR8', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LGR8", "wwn-0x5000c500918e79d4"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LGR8-part1", "wwn-0x5000c500918e79d4-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LGR8", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918e79d4"}}, "skip_reason": 2021-04-04T09:26:16.216 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-04-04T09:26:17.305 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-8z0UVl-A1HJ-CDjR-t6Cy-akYj-v9RE-RjqzS0', 'nvme-INTEL_SSDPEDMD400G4_PHFT6204010U400BGN', 'nvme-nvme.8086-50484654363230343031305534303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.148193", "end": "2021-04-04 09:26:17.608882", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-8z0UVl-A1HJ-CDjR-t6Cy-akYj-v9RE-RjqzS0", "nvme-INTEL_SSDPEDMD400G4_PHFT6204010U400BGN", "nvme-nvme.8086-50484654363230343031305534303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:16.460689", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-04-04T09:26:17.307 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:17.319 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:17.539 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-ptycqE-zSu9-vcns-cHKI-NzMV-eslB-UlIr54', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400QR400BGN', 'nvme-nvme.8086-50484654363230343030515234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.197611", "end": "2021-04-04 09:26:17.506513", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ptycqE-zSu9-vcns-cHKI-NzMV-eslB-UlIr54", "nvme-INTEL_SSDPEDMD400G4_PHFT620400QR400BGN", "nvme-nvme.8086-50484654363230343030515234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:16.308902", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-04-04T09:26:17.541 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-04T09:26:17.550 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:17.551 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:18.597 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r97TgoRh4cj8epOi8I3KC3RdYTd1V3PlJw'], 'uuids': ['eda2d963-7e08-4799-9d1c-a136a4937ff5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.157698", "end": "2021-04-04 09:26:18.904595", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r97TgoRh4cj8epOi8I3KC3RdYTd1V3PlJw"], "labels": [], "masters": [], "uuids": ["eda2d963-7e08-4799-9d1c-a136a4937ff5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:17.746897", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-04T09:26:18.598 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:18.609 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:18.818 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAlohrJ6VzemIl5Qe2JhSCZObQQlgBK8tKGr'], 'uuids': ['d63bd2ab-2d42-4856-8839-3d460cff6bb4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.138007", "end": "2021-04-04 09:26:18.791936", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAlohrJ6VzemIl5Qe2JhSCZObQQlgBK8tKGr"], "labels": [], "masters": [], "uuids": ["d63bd2ab-2d42-4856-8839-3d460cff6bb4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:17.653929", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-04T09:26:18.833 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:19.856 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r9AOFFm0vaDkEWynVqlcV17klRs7H64LYJ'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.120035", "end": "2021-04-04 09:26:20.170114", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r9AOFFm0vaDkEWynVqlcV17klRs7H64LYJ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:19.050079", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-04T09:26:19.867 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:20.057 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAlokEUyMhDgPyRcdOsyZf5NKE50nvSKrEQb'], 'uuids': ['7ff5fadd-3656-4f62-9c7f-56bf5830cde2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.101176", "end": "2021-04-04 09:26:20.033492", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAlokEUyMhDgPyRcdOsyZf5NKE50nvSKrEQb"], "labels": [], "masters": [], "uuids": ["7ff5fadd-3656-4f62-9c7f-56bf5830cde2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:18.932316", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-04T09:26:20.070 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:21.105 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r9YUwtOamvxvI1ZGqLPSqMEcofQr2M0THT'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.123980", "end": "2021-04-04 09:26:21.420656", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r9YUwtOamvxvI1ZGqLPSqMEcofQr2M0THT"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:20.296676", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-04T09:26:21.138 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:21 +0000 (0:00:08.024) 0:02:18.156 ********** 2021-04-04T09:26:21.228 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-04-04T09:26:21.230 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:21.293 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-04-04T09:26:21.294 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAloN2xRL43pmbeDcu8lqC9PMq8d2g6IDuU1'], 'uuids': ['c4bf253f-3177-479b-846b-fbd6187d422c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.104585", "end": "2021-04-04 09:26:21.268326", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAloN2xRL43pmbeDcu8lqC9PMq8d2g6IDuU1"], "labels": [], "masters": [], "uuids": ["c4bf253f-3177-479b-846b-fbd6187d422c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:20.163741", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-04-04T09:26:21.333 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:21 +0000 (0:00:00.194) 0:02:18.351 ********** 2021-04-04T09:26:21.454 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-04-04T09:26:21.456 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:21.509 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r93rOKGkskRR3v3Ws28vJRKw6vgarC3k6y'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.144919", "end": "2021-04-04 09:26:21.817282", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r93rOKGkskRR3v3Ws28vJRKw6vgarC3k6y"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:21.672363", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142356 s, 810 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142356 s, 810 MB/s"], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:21.522 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:21.534 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:21.755 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAloLUMJh7GPEpABmNY4zJrf7hFhfEUKzMHb'], 'uuids': ['b08d2f36-660e-43b9-acbe-256e7495825d']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.161002", "end": "2021-04-04 09:26:21.728879", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAloLUMJh7GPEpABmNY4zJrf7hFhfEUKzMHb"], "labels": [], "masters": [], "uuids": ["b08d2f36-660e-43b9-acbe-256e7495825d"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:21.567877", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.156443 s, 737 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.156443 s, 737 MB/s"], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:21.767 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:21.769 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:21.783 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r9qgWll2yTcDOFPlBeycdKGESUvRDUVf0P'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.144951", "end": "2021-04-04 09:26:22.092042", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r9qgWll2yTcDOFPlBeycdKGESUvRDUVf0P"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:21.947091", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140511 s, 821 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140511 s, 821 MB/s"], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:21.795 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5L5J2', 'wwn-0x5000c500918e77a4'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5L5J2-part1', 'wwn-0x5000c500918e77a4-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918e77a4', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5L5J2', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L5J2", "wwn-0x5000c500918e77a4"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L5J2-part1", "wwn-0x5000c500918e77a4-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5L5J2", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918e77a4"}}, "skip_reason": 2021-04-04T09:26:21.797 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-04-04T09:26:21.965 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-8z0UVl-A1HJ-CDjR-t6Cy-akYj-v9RE-RjqzS0', 'nvme-INTEL_SSDPEDMD400G4_PHFT6204010U400BGN', 'nvme-nvme.8086-50484654363230343031305534303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.048030", "end": "2021-04-04 09:26:22.275424", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-8z0UVl-A1HJ-CDjR-t6Cy-akYj-v9RE-RjqzS0", "nvme-INTEL_SSDPEDMD400G4_PHFT6204010U400BGN", "nvme-nvme.8086-50484654363230343031305534303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:22.227394", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0438293 s, 2.6 GB/s", "std 2021-04-04T09:26:21.967 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0438293 s, 2.6 GB/s"], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:21.978 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:21.980 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:22.075 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAlo93PXdF3Fj9MfDPLRM8HZLDA1xl5HcFX6'], 'uuids': ['dc850323-c9d1-430f-8004-e8d25d6e1dde']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.170053", "end": "2021-04-04 09:26:22.042852", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAlo93PXdF3Fj9MfDPLRM8HZLDA1xl5HcFX6"], "labels": [], "masters": [], "uuids": ["dc850323-c9d1-430f-8004-e8d25d6e1dde"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:21.872799", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.165333 s, 698 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.165333 s, 698 MB/s"], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:22.078 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LGR8', 'wwn-0x5000c500918e79d4'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LGR8-part1', 'wwn-0x5000c500918e79d4-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500918e79d4', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W5LGR8', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LGR8", "wwn-0x5000c500918e79d4"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LGR8-part1", "wwn-0x5000c500918e79d4-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LGR8", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918e79d4"}}, "skip_reason": 2021-04-04T09:26:22.080 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-04-04T09:26:22.239 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r97TgoRh4cj8epOi8I3KC3RdYTd1V3PlJw'], 'uuids': ['eda2d963-7e08-4799-9d1c-a136a4937ff5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.134432", "end": "2021-04-04 09:26:22.550629", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r97TgoRh4cj8epOi8I3KC3RdYTd1V3PlJw"], "labels": [], "masters": [], "uuids": ["eda2d963-7e08-4799-9d1c-a136a4937ff5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:22.416197", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.132009 s, 874 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.132009 s, 874 MB/s"], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:22.240 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:22.252 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:22.264 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-ptycqE-zSu9-vcns-cHKI-NzMV-eslB-UlIr54', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400QR400BGN', 'nvme-nvme.8086-50484654363230343030515234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.051713", "end": "2021-04-04 09:26:22.230322", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-ptycqE-zSu9-vcns-cHKI-NzMV-eslB-UlIr54", "nvme-INTEL_SSDPEDMD400G4_PHFT620400QR400BGN", "nvme-nvme.8086-50484654363230343030515234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:22.178609", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0470408 s, 2.5 GB/s", "std 2021-04-04T09:26:22.266 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0470408 s, 2.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:22.266 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:22.278 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:22.542 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r9AOFFm0vaDkEWynVqlcV17klRs7H64LYJ'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.158369", "end": "2021-04-04 09:26:22.850164", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r9AOFFm0vaDkEWynVqlcV17klRs7H64LYJ"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:22.691795", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153757 s, 750 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153757 s, 750 MB/s"], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:22.544 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:22.558 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAlohrJ6VzemIl5Qe2JhSCZObQQlgBK8tKGr'], 'uuids': ['d63bd2ab-2d42-4856-8839-3d460cff6bb4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.157575", "end": "2021-04-04 09:26:22.530546", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAlohrJ6VzemIl5Qe2JhSCZObQQlgBK8tKGr"], "labels": [], "masters": [], "uuids": ["d63bd2ab-2d42-4856-8839-3d460cff6bb4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:22.372971", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.154612 s, 746 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.154612 s, 746 MB/s"], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:22.570 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:22.584 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:22.827 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r9YUwtOamvxvI1ZGqLPSqMEcofQr2M0THT'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.152661", "end": "2021-04-04 09:26:23.144675", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-UP2X3WK3TaId9oknhIUSv2Xs4CzhD5r9YUwtOamvxvI1ZGqLPSqMEcofQr2M0THT"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:22.992014", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148034 s, 779 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148034 s, 779 MB/s"], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:22.862 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:22 +0000 (0:00:01.529) 0:02:19.880 ********** 2021-04-04T09:26:22.879 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAlokEUyMhDgPyRcdOsyZf5NKE50nvSKrEQb'], 'uuids': ['7ff5fadd-3656-4f62-9c7f-56bf5830cde2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.163995", "end": "2021-04-04 09:26:22.848339", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAlokEUyMhDgPyRcdOsyZf5NKE50nvSKrEQb"], "labels": [], "masters": [], "uuids": ["7ff5fadd-3656-4f62-9c7f-56bf5830cde2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:22.684344", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.159221 s, 724 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.159221 s, 724 MB/s"], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:22.880 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:23.176 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAloN2xRL43pmbeDcu8lqC9PMq8d2g6IDuU1'], 'uuids': ['c4bf253f-3177-479b-846b-fbd6187d422c']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.159695", "end": "2021-04-04 09:26:23.147827", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-2yXRAF0aIhSY2w4FBM58fB118Vo2yAloN2xRL43pmbeDcu8lqC9PMq8d2g6IDuU1"], "labels": [], "masters": [], "uuids": ["c4bf253f-3177-479b-846b-fbd6187d422c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-04-04 09:26:22.988132", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.15489 s, 745 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.15489 s, 745 MB/s"], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:23.192 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-04-04T09:26:23.193 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.126073", "end": "2021-04-04 09:26:23.448826", "rc": 0, "start": "2021-04-04 09:26:23.322753", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:23.216 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:23 +0000 (0:00:00.353) 0:02:20.234 ********** 2021-04-04T09:26:23.242 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:23 +0000 (0:00:00.026) 0:02:20.260 ********** 2021-04-04T09:26:23.439 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-04-04T09:26:23.441 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.052924", "end": "2021-04-04 09:26:23.751564", "rc": 0, "start": "2021-04-04 09:26:23.698640", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:23.463 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:23 +0000 (0:00:00.220) 0:02:20.481 ********** 2021-04-04T09:26:23.532 INFO:teuthology.task.ansible.out: 2021-04-04T09:26:23.534 INFO:teuthology.task.ansible.out:TASK [testnode : Remove physical volumes] ************************************** 2021-04-04T09:26:23.611 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:23 +0000 (0:00:00.147) 0:02:20.628 ********** 2021-04-04T09:26:23.618 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-04-04T09:26:23.620 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.141498", "end": "2021-04-04 09:26:23.478394", "rc": 0, "start": "2021-04-04 09:26:23.336896", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:23.646 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:23 +0000 (0:00:00.034) 0:02:20.663 ********** 2021-04-04T09:26:23.653 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:23.682 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:23 +0000 (0:00:00.036) 0:02:20.700 ********** 2021-04-04T09:26:23.709 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-04-04T09:26:23.711 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:23.739 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:23 +0000 (0:00:00.056) 0:02:20.757 ********** 2021-04-04T09:26:23.765 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-04-04T09:26:23.767 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:23.793 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:23 +0000 (0:00:00.053) 0:02:20.810 ********** 2021-04-04T09:26:23.820 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-04-04T09:26:23.821 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:23.846 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:23 +0000 (0:00:00.052) 0:02:20.863 ********** 2021-04-04T09:26:23.864 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-04-04T09:26:23.866 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.074109", "end": "2021-04-04 09:26:23.831791", "rc": 0, "start": "2021-04-04 09:26:23.757682", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:23.892 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:23 +0000 (0:00:00.046) 0:02:20.910 ********** 2021-04-04T09:26:23.919 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-04-04T09:26:23.953 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:23 +0000 (0:00:00.061) 0:02:20.971 ********** 2021-04-04T09:26:23.982 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-04-04T09:26:23.984 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.010 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.056) 0:02:21.027 ********** 2021-04-04T09:26:24.017 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-04-04T09:26:24.019 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-04T09:26:24.042 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.032) 0:02:21.060 ********** 2021-04-04T09:26:24.050 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-04-04T09:26:24.052 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.081 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.038) 0:02:21.098 ********** 2021-04-04T09:26:24.087 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-04-04T09:26:24.089 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'block_used': 13760082, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226335026, 'size_available': 927068266496, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149513, 'block_size': 4096, 'inode_available': 60905463}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226335026, "block_size": 4096, "block_total": 240095108, "block_used": 13760082, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905463, "inode_total": 61054976, "inode_used": 149513, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927068266496, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.129 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.048) 0:02:21.146 ********** 2021-04-04T09:26:24.137 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-04-04T09:26:24.138 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.174 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.045) 0:02:21.192 ********** 2021-04-04T09:26:24.181 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-04-04T09:26:24.183 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.207 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.033) 0:02:21.225 ********** 2021-04-04T09:26:24.213 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-04-04T09:26:24.215 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.239 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.032) 0:02:21.257 ********** 2021-04-04T09:26:24.247 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-04T09:26:24.248 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.277 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.037) 0:02:21.295 ********** 2021-04-04T09:26:24.305 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-04T09:26:24.306 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.335 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.058) 0:02:21.353 ********** 2021-04-04T09:26:24.362 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-04-04T09:26:24.363 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.385 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.049) 0:02:21.403 ********** 2021-04-04T09:26:24.391 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-04-04T09:26:24.393 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-04-04T09:26:24.416 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.031) 0:02:21.434 ********** 2021-04-04T09:26:24.422 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-04-04T09:26:24.424 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.449 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.032) 0:02:21.467 ********** 2021-04-04T09:26:24.455 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-04-04T09:26:24.457 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'block_used': 13760082, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226335026, 'size_available': 927068266496, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149513, 'block_size': 4096, 'inode_available': 60905463}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226335026, "block_size": 4096, "block_total": 240095108, "block_used": 13760082, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905463, "inode_total": 61054976, "inode_used": 149513, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927068266496, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.492 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.043) 0:02:21.510 ********** 2021-04-04T09:26:24.499 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-04-04T09:26:24.501 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.525 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.032) 0:02:21.543 ********** 2021-04-04T09:26:24.531 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-04-04T09:26:24.533 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.564 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.038) 0:02:21.582 ********** 2021-04-04T09:26:24.592 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-04T09:26:24.593 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.616 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.051) 0:02:21.633 ********** 2021-04-04T09:26:24.642 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-04T09:26:24.644 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.670 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.054) 0:02:21.688 ********** 2021-04-04T09:26:24.707 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-04-04T09:26:24.709 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.737 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.066) 0:02:21.754 ********** 2021-04-04T09:26:24.766 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-04-04T09:26:24.768 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.790 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.053) 0:02:21.808 ********** 2021-04-04T09:26:24.817 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-04-04T09:26:24.819 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:24.841 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:24 +0000 (0:00:00.050) 0:02:21.859 ********** 2021-04-04T09:26:24.982 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-04-04T09:26:24.983 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-04-04T09:26:25.011 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:25 +0000 (0:00:00.169) 0:02:22.028 ********** 2021-04-04T09:26:25.180 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-04-04T09:26:25.205 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:25 +0000 (0:00:00.194) 0:02:22.223 ********** 2021-04-04T09:26:25.485 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-04-04T09:26:25.486 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-04T09:26:25.508 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-04T09:26:25.709 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-04T09:26:25.773 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-04T09:26:25.965 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-04T09:26:26.018 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-04T09:26:26.188 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-04T09:26:26.242 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-04T09:26:26.244 INFO:teuthology.task.ansible.out: 2021-04-04T09:26:26.410 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-04T09:26:26.436 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:26 +0000 (0:00:01.231) 0:02:23.454 ********** 2021-04-04T09:26:26.484 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-04-04T09:26:26.512 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:26 +0000 (0:00:00.075) 0:02:23.529 ********** 2021-04-04T09:26:26.581 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-04-04T09:26:26.583 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-04-04T09:26:26.611 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:26 +0000 (0:00:00.099) 0:02:23.629 ********** 2021-04-04T09:26:26.660 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-04-04T09:26:26.687 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:26 +0000 (0:00:00.075) 0:02:23.705 ********** 2021-04-04T09:26:26.767 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-04-04T09:26:26.769 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-04-04T09:26:26.865 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-04-04T09:26:26.892 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-04-04T09:26:26.988 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-04-04T09:26:27.022 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-04-04T09:26:27.109 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-04-04T09:26:27.142 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-04-04T09:26:27.154 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:27.185 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:27 +0000 (0:00:00.498) 0:02:24.203 ********** 2021-04-04T09:26:27.213 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-04-04T09:26:27.215 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:27.239 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:27 +0000 (0:00:00.053) 0:02:24.256 ********** 2021-04-04T09:26:27.247 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-04-04T09:26:27.248 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-04-04T09:26:27.250 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:27.292 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:27 +0000 (0:00:00.053) 0:02:24.309 ********** 2021-04-04T09:26:27.300 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-04-04T09:26:27.302 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:27.327 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:27 +0000 (0:00:00.035) 0:02:24.345 ********** 2021-04-04T09:26:27.334 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-04-04T09:26:27.335 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:27.368 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:27 +0000 (0:00:00.040) 0:02:24.386 ********** 2021-04-04T09:26:27.397 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-04-04T09:26:27.399 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:27.424 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:27 +0000 (0:00:00.056) 0:02:24.442 ********** 2021-04-04T09:26:27.475 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-04-04T09:26:27.477 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-04-04T09:26:27.500 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:27 +0000 (0:00:00.075) 0:02:24.518 ********** 2021-04-04T09:26:27.538 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-04-04T09:26:27.539 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-04-04T09:26:27.563 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:27 +0000 (0:00:00.062) 0:02:24.581 ********** 2021-04-04T09:26:27.580 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-04-04T09:26:27.582 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-04-04T09:26:27.616 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:27 +0000 (0:00:00.052) 0:02:24.634 ********** 2021-04-04T09:26:27.665 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-04-04T09:26:27.667 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-04-04T09:26:27.690 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:27 +0000 (0:00:00.074) 0:02:24.708 ********** 2021-04-04T09:26:28.071 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-04-04T09:26:28.073 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true} 2021-04-04T09:26:28.096 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:28 +0000 (0:00:00.406) 0:02:25.114 ********** 2021-04-04T09:26:28.104 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true} 2021-04-04T09:26:28.131 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:28 +0000 (0:00:00.035) 0:02:25.149 ********** 2021-04-04T09:26:28.576 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-04-04T09:26:28.600 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:28 +0000 (0:00:00.468) 0:02:25.617 ********** 2021-04-04T09:26:28.609 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-04-04T09:26:28.643 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:28 +0000 (0:00:00.043) 0:02:25.661 ********** 2021-04-04T09:26:28.653 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-04-04T09:26:28.655 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:28.681 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:28 +0000 (0:00:00.037) 0:02:25.699 ********** 2021-04-04T09:26:28.691 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:28.714 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:28 +0000 (0:00:00.033) 0:02:25.732 ********** 2021-04-04T09:26:28.723 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-04-04T09:26:28.725 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:28.758 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:28 +0000 (0:00:00.043) 0:02:25.776 ********** 2021-04-04T09:26:28.766 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:28.792 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:28 +0000 (0:00:00.033) 0:02:25.810 ********** 2021-04-04T09:26:28.799 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-04-04T09:26:28.801 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:28.830 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:28 +0000 (0:00:00.037) 0:02:25.848 ********** 2021-04-04T09:26:28.839 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:28.863 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:28 +0000 (0:00:00.033) 0:02:25.881 ********** 2021-04-04T09:26:28.870 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-04-04T09:26:28.871 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:28.895 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:28 +0000 (0:00:00.032) 0:02:25.913 ********** 2021-04-04T09:26:28.902 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:28.939 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:28 +0000 (0:00:00.043) 0:02:25.956 ********** 2021-04-04T09:26:29.059 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-04-04T09:26:29.061 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-04-04T09:26:29.085 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:29 +0000 (0:00:00.146) 0:02:26.103 ********** 2021-04-04T09:26:29.103 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-04-04T09:26:29.128 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:29 +0000 (0:00:00.042) 0:02:26.146 ********** 2021-04-04T09:26:29.219 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-04-04T09:26:29.222 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-04-04T09:26:29.248 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:29 +0000 (0:00:00.120) 0:02:26.266 ********** 2021-04-04T09:26:29.275 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-04-04T09:26:29.303 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:29 +0000 (0:00:00.054) 0:02:26.321 ********** 2021-04-04T09:26:29.620 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-04-04T09:26:29.623 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-04-04T09:26:29.656 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:29 +0000 (0:00:00.352) 0:02:26.674 ********** 2021-04-04T09:26:29.685 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-04-04T09:26:29.717 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:29 +0000 (0:00:00.061) 0:02:26.735 ********** 2021-04-04T09:26:30.019 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-04-04T09:26:30.021 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-04-04T09:26:30.050 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:30 +0000 (0:00:00.332) 0:02:27.068 ********** 2021-04-04T09:26:30.073 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-04-04T09:26:30.099 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:30 +0000 (0:00:00.048) 0:02:27.117 ********** 2021-04-04T09:26:30.969 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-04-04T09:26:30.970 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1617528026, "cache_updated": false, "changed": false} 2021-04-04T09:26:31.006 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:31 +0000 (0:00:00.906) 0:02:28.023 ********** 2021-04-04T09:26:31.057 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"cache_update_time": 1617528070, "cache_updated": false, "changed": false} 2021-04-04T09:26:31.082 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:31 +0000 (0:00:00.076) 0:02:28.100 ********** 2021-04-04T09:26:31.204 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-04-04T09:26:31.205 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.045270", "end": "2021-04-04 09:26:31.174923", "rc": 0, "start": "2021-04-04 09:26:31.129653", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-04-04T09:26:31.238 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:31 +0000 (0:00:00.155) 0:02:28.256 ********** 2021-04-04T09:26:31.266 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-04-04T09:26:31.268 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:31.302 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:31 +0000 (0:00:00.063) 0:02:28.320 ********** 2021-04-04T09:26:31.309 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-04-04T09:26:31.312 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.049453", "end": "2021-04-04 09:26:31.591756", "rc": 0, "start": "2021-04-04 09:26:31.542303", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-04-04T09:26:31.337 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:31 +0000 (0:00:00.034) 0:02:28.355 ********** 2021-04-04T09:26:31.343 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-04-04T09:26:31.345 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:31.368 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:31 +0000 (0:00:00.031) 0:02:28.386 ********** 2021-04-04T09:26:31.374 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-04-04T09:26:31.376 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:31.400 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:31 +0000 (0:00:00.031) 0:02:28.418 ********** 2021-04-04T09:26:31.426 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-04-04T09:26:31.428 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:31.452 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:31 +0000 (0:00:00.052) 0:02:28.470 ********** 2021-04-04T09:26:32.245 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-04-04T09:26:32.247 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1617528026, "cache_updated": false, "changed": false} 2021-04-04T09:26:32.274 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:32 +0000 (0:00:00.821) 0:02:29.291 ********** 2021-04-04T09:26:32.303 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-04-04T09:26:32.305 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:32.329 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:32 +0000 (0:00:00.055) 0:02:29.347 ********** 2021-04-04T09:26:32.337 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-04-04T09:26:32.339 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"cache_update_time": 1617528070, "cache_updated": false, "changed": false} 2021-04-04T09:26:32.367 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:32 +0000 (0:00:00.037) 0:02:29.384 ********** 2021-04-04T09:26:32.393 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-04-04T09:26:32.396 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:32.418 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:32 +0000 (0:00:00.051) 0:02:29.435 ********** 2021-04-04T09:26:32.666 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-04-04T09:26:32.667 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-04-04T09:26:32.693 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:32 +0000 (0:00:00.275) 0:02:29.711 ********** 2021-04-04T09:26:32.727 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-04-04T09:26:32.729 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:32.752 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:32 +0000 (0:00:00.058) 0:02:29.769 ********** 2021-04-04T09:26:32.768 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-04-04T09:26:32.770 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-04-04T09:26:32.795 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:32 +0000 (0:00:00.043) 0:02:29.813 ********** 2021-04-04T09:26:32.824 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-04-04T09:26:32.826 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:32.848 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:32 +0000 (0:00:00.052) 0:02:29.866 ********** 2021-04-04T09:26:32.971 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-04-04T09:26:32.972 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2021-04-04 09:23:42 UTC", "ActiveEnterTimestampMonotonic": "26237815", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount systemd-tmpfiles-setup.service sysinit.target system.slice systemd-journald.socket basic.target network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2021-04-04 09:23:35 UTC", "AssertTimestampMonotonic": "19628171", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2021-04-04 09:23:35 UTC", "ConditionTimestampMonotonic": "19628170", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-04-04T09:26:32.974 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1978", "ExecMainStartTimestamp": "Sun 2021-04-04 09:23:42 UTC", "ExecMainStartTimestampMonotonic": "26237765", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2021-04-04 09:23:35 UTC", "InactiveExitTimestampMonotonic": "19629017", "InvocationID": "b66007798a1f47d6a6af9153f829f0d7", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128028", "LimitNPROCSoft": "128028", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-04-04T09:26:32.976 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128028", "LimitSIGPENDINGSoft": "128028", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1978", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2021-04-04 09:23:42 UTC", "S 2021-04-04T09:26:32.977 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "26237815", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2021-04-04 09:23:42 UTC", "WatchdogTimestampMonotonic": "26237812", "WatchdogUSec": "0"}} 2021-04-04T09:26:33.003 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:33 +0000 (0:00:00.155) 0:02:30.021 ********** 2021-04-04T09:26:33.032 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-04-04T09:26:33.057 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:33 +0000 (0:00:00.054) 0:02:30.075 ********** 2021-04-04T09:26:33.066 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-04-04T09:26:33.067 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Sun 2021-04-04 09:23:51 UTC", "ActiveEnterTimestampMonotonic": "26116584", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target basic.target systemd-journald.socket -.mount system.slice sysinit.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Sun 2021-04-04 09:23:45 UTC", "AssertTimestampMonotonic": "19604048", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Sun 2021-04-04 09:23:45 UTC", "ConditionTimestampMonotonic": "19604048", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-04-04T09:26:33.069 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1838", "ExecMainStartTimestamp": "Sun 2021-04-04 09:23:51 UTC", "ExecMainStartTimestampMonotonic": "26116569", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Sun 2021-04-04 09:23:45 UTC", "InactiveExitTimestampMonotonic": "19604830", "InvocationID": "63d3d26a072b4c59926d545ce17b7ef7", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128028", "LimitNPROCSoft": "128028", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-04-04T09:26:33.071 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128028", "LimitSIGPENDINGSoft": "128028", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1838", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2021-04-04 09:23:51 UTC", "S 2021-04-04T09:26:33.073 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "26116584", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Sun 2021-04-04 09:23:51 UTC", "WatchdogTimestampMonotonic": "26116582", "WatchdogUSec": "0"}} 2021-04-04T09:26:33.096 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:33 +0000 (0:00:00.038) 0:02:30.113 ********** 2021-04-04T09:26:33.101 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-04T09:26:33.103 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:33.136 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:33 +0000 (0:00:00.040) 0:02:30.154 ********** 2021-04-04T09:26:33.145 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-04-04T09:26:33.172 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:33 +0000 (0:00:00.036) 0:02:30.190 ********** 2021-04-04T09:26:33.179 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-04-04T09:26:33.181 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:33.205 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:33 +0000 (0:00:00.032) 0:02:30.223 ********** 2021-04-04T09:26:33.212 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-04T09:26:33.213 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:33.239 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:33 +0000 (0:00:00.033) 0:02:30.256 ********** 2021-04-04T09:26:33.279 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-04-04T09:26:33.280 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:33.308 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:33 +0000 (0:00:00.068) 0:02:30.325 ********** 2021-04-04T09:26:34.071 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-04-04T09:26:34.072 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1617528026, "cache_updated": false, "changed": false} 2021-04-04T09:26:34.095 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:34 +0000 (0:00:00.787) 0:02:31.113 ********** 2021-04-04T09:26:34.122 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-04-04T09:26:34.124 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:34.145 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:34 +0000 (0:00:00.049) 0:02:31.162 ********** 2021-04-04T09:26:34.255 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-04-04T09:26:34.257 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"cache_update_time": 1617528070, "cache_updated": false, "changed": false} 2021-04-04T09:26:34.295 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:34 +0000 (0:00:00.149) 0:02:31.312 ********** 2021-04-04T09:26:34.304 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-04-04T09:26:34.306 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-04-04T09:26:34.330 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:34 +0000 (0:00:00.035) 0:02:31.348 ********** 2021-04-04T09:26:34.337 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-04-04T09:26:34.339 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:34.369 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:34 +0000 (0:00:00.038) 0:02:31.386 ********** 2021-04-04T09:26:34.524 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-04-04T09:26:34.526 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-04-04T09:26:34.549 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:34 +0000 (0:00:00.180) 0:02:31.567 ********** 2021-04-04T09:26:34.662 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-04-04T09:26:34.663 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-04-04T09:26:34.700 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:34 +0000 (0:00:00.150) 0:02:31.718 ********** 2021-04-04T09:26:34.903 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-04-04T09:26:34.930 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:34 +0000 (0:00:00.230) 0:02:31.948 ********** 2021-04-04T09:26:35.780 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-04-04T09:26:35.781 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-04-04T09:26:35.782 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-04-04T09:26:35.818 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:35 +0000 (0:00:00.887) 0:02:32.835 ********** 2021-04-04T09:26:35.840 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:35 +0000 (0:00:00.021) 0:02:32.857 ********** 2021-04-04T09:26:36.341 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-04-04T09:26:36.343 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false} ok: [smithi078.front.sepia.ceph.com] => {"changed": false} 2021-04-04T09:26:36.382 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:36 +0000 (0:00:00.541) 0:02:33.399 ********** 2021-04-04T09:26:36.405 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:36 +0000 (0:00:00.023) 0:02:33.423 ********** 2021-04-04T09:26:36.423 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-04-04T09:26:36.425 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-04-04T09:26:36.448 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:36 +0000 (0:00:00.043) 0:02:33.466 ********** 2021-04-04T09:26:36.458 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-04-04T09:26:36.485 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:36 +0000 (0:00:00.036) 0:02:33.502 ********** 2021-04-04T09:26:36.597 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-04-04T09:26:36.600 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1617527998.1519501, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1617527998.14795, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 262873, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1617527998.14795, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3476867264", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-04-04T09:26:36.625 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:36 +0000 (0:00:00.139) 0:02:33.642 ********** 2021-04-04T09:26:36.643 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1617528010.0722, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1617528010.0682, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 262873, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1617528010.0682, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3476867264", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-04-04T09:26:36.671 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:36 +0000 (0:00:00.046) 0:02:33.689 ********** 2021-04-04T09:26:37.015 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-04-04T09:26:37.017 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-04-04T09:26:37.017 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-04-04T09:26:37.042 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.370) 0:02:34.060 ********** 2021-04-04T09:26:37.063 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.021) 0:02:34.081 ********** 2021-04-04T09:26:37.080 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-04-04T09:26:37.082 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-04-04T09:26:37.104 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.040) 0:02:34.122 ********** 2021-04-04T09:26:37.111 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-04-04T09:26:37.136 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.032) 0:02:34.154 ********** 2021-04-04T09:26:37.142 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-04-04T09:26:37.144 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.168 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.031) 0:02:34.186 ********** 2021-04-04T09:26:37.175 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.206 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.037) 0:02:34.224 ********** 2021-04-04T09:26:37.296 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-04-04T09:26:37.298 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-04-04T09:26:37.324 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.117) 0:02:34.342 ********** 2021-04-04T09:26:37.342 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-04-04T09:26:37.375 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.050) 0:02:34.393 ********** 2021-04-04T09:26:37.466 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-04-04T09:26:37.467 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-04-04T09:26:37.492 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.117) 0:02:34.510 ********** 2021-04-04T09:26:37.509 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-04-04T09:26:37.534 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.042) 0:02:34.552 ********** 2021-04-04T09:26:37.541 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-04-04T09:26:37.543 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.568 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.033) 0:02:34.586 ********** 2021-04-04T09:26:37.575 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.601 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.032) 0:02:34.618 ********** 2021-04-04T09:26:37.608 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-04-04T09:26:37.610 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.619 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.621 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.631 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.660 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.058) 0:02:34.677 ********** 2021-04-04T09:26:37.668 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.670 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.700 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.040) 0:02:34.718 ********** 2021-04-04T09:26:37.709 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-04-04T09:26:37.710 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.712 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.724 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.726 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi078.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.760 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.059) 0:02:34.777 ********** 2021-04-04T09:26:37.767 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.770 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi122.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.770 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.772 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.796 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.036) 0:02:34.814 ********** 2021-04-04T09:26:37.803 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-04-04T09:26:37.804 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.830 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.033) 0:02:34.848 ********** 2021-04-04T09:26:37.839 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.875 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.044) 0:02:34.892 ********** 2021-04-04T09:26:37.884 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-04-04T09:26:37.886 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.911 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.036) 0:02:34.929 ********** 2021-04-04T09:26:37.918 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.948 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.036) 0:02:34.965 ********** 2021-04-04T09:26:37.957 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-04-04T09:26:37.958 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:37.990 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:37 +0000 (0:00:00.042) 0:02:35.008 ********** 2021-04-04T09:26:37.998 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:38.026 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:38 +0000 (0:00:00.035) 0:02:35.044 ********** 2021-04-04T09:26:38.032 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-04-04T09:26:38.034 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:38.058 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:38 +0000 (0:00:00.031) 0:02:35.075 ********** 2021-04-04T09:26:38.064 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:38.090 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:38 +0000 (0:00:00.032) 0:02:35.108 ********** 2021-04-04T09:26:38.096 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-04-04T09:26:38.098 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:38.124 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:38 +0000 (0:00:00.033) 0:02:35.141 ********** 2021-04-04T09:26:38.130 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:38.158 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:38 +0000 (0:00:00.034) 0:02:35.176 ********** 2021-04-04T09:26:38.164 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-04-04T09:26:38.165 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:38.201 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:38 +0000 (0:00:00.043) 0:02:35.219 ********** 2021-04-04T09:26:38.209 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:38.240 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:38 +0000 (0:00:00.038) 0:02:35.257 ********** 2021-04-04T09:26:38.246 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-04-04T09:26:38.248 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:38.277 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:38 +0000 (0:00:00.036) 0:02:35.294 ********** 2021-04-04T09:26:38.285 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:38.318 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:38 +0000 (0:00:00.041) 0:02:35.336 ********** 2021-04-04T09:26:38.326 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-04-04T09:26:38.329 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:38.357 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:38 +0000 (0:00:00.038) 0:02:35.375 ********** 2021-04-04T09:26:38.363 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:38.392 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:38 +0000 (0:00:00.035) 0:02:35.410 ********** 2021-04-04T09:26:38.523 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-04-04T09:26:38.525 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011134", "end": "2021-04-04 09:26:38.492569", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-04-04 09:26:38.481435", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:38.578 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010917", "end": "2021-04-04 09:26:38.868195", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-04-04 09:26:38.857278", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:38.642 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005699", "end": "2021-04-04 09:26:38.623484", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-04-04 09:26:38.617785", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:38.717 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005838", "end": "2021-04-04 09:26:39.014147", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-04-04 09:26:39.008309", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:38.760 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005574", "end": "2021-04-04 09:26:38.737358", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-04-04 09:26:38.731784", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:38.854 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005449", "end": "2021-04-04 09:26:39.157970", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-04-04 09:26:39.152521", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:38.907 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.022589", "end": "2021-04-04 09:26:38.874293", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-04-04 09:26:38.851704", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:38.935 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:38 +0000 (0:00:00.542) 0:02:35.952 ********** 2021-04-04T09:26:38.962 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-04-04T09:26:38.990 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:38 +0000 (0:00:00.055) 0:02:36.008 ********** 2021-04-04T09:26:39.000 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-04-04T09:26:39.002 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.006485", "end": "2021-04-04 09:26:39.274827", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-04-04 09:26:39.268342", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:39.039 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:39 +0000 (0:00:00.048) 0:02:36.056 ********** 2021-04-04T09:26:39.065 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-04-04T09:26:39.089 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:39 +0000 (0:00:00.050) 0:02:36.107 ********** 2021-04-04T09:26:39.138 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-04-04T09:26:39.139 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-04-04T09:26:39.221 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-04-04T09:26:39.294 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-04-04T09:26:39.329 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:39 +0000 (0:00:00.239) 0:02:36.347 ********** 2021-04-04T09:26:39.366 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:39 +0000 (0:00:00.036) 0:02:36.383 ********** 2021-04-04T09:26:40.071 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-04-04T09:26:40.073 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] 2021-04-04T09:26:40.112 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] 2021-04-04T09:26:40.145 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:40 +0000 (0:00:00.779) 0:02:37.163 ********** 2021-04-04T09:26:40.195 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:40 +0000 (0:00:00.049) 0:02:37.212 ********** 2021-04-04T09:26:40.201 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-04-04T09:26:40.203 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-04-04T09:26:40.227 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:40 +0000 (0:00:00.032) 0:02:37.245 ********** 2021-04-04T09:26:40.234 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-04-04T09:26:40.262 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:40 +0000 (0:00:00.034) 0:02:37.280 ********** 2021-04-04T09:26:40.268 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-04-04T09:26:40.270 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:40.293 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:40 +0000 (0:00:00.030) 0:02:37.311 ********** 2021-04-04T09:26:40.300 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:40.326 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:40 +0000 (0:00:00.033) 0:02:37.344 ********** 2021-04-04T09:26:40.343 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-04-04T09:26:40.345 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-04-04T09:26:40.378 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:40 +0000 (0:00:00.051) 0:02:37.395 ********** 2021-04-04T09:26:40.386 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-04-04T09:26:40.425 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:40 +0000 (0:00:00.047) 0:02:37.443 ********** 2021-04-04T09:26:41.282 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-04-04T09:26:41.284 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"cache_update_time": 1617528070, "cache_updated": false, "changed": false} 2021-04-04T09:26:41.313 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:41 +0000 (0:00:00.888) 0:02:38.331 ********** 2021-04-04T09:26:41.331 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"cache_update_time": 1617528026, "cache_updated": false, "changed": false} 2021-04-04T09:26:41.355 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:41 +0000 (0:00:00.041) 0:02:38.373 ********** 2021-04-04T09:26:41.362 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-04-04T09:26:41.364 INFO:teuthology.task.ansible.out:skipping: [smithi122.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:41.390 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:41 +0000 (0:00:00.034) 0:02:38.408 ********** 2021-04-04T09:26:41.396 INFO:teuthology.task.ansible.out:skipping: [smithi078.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-04-04T09:26:41.425 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:41 +0000 (0:00:00.034) 0:02:38.442 ********** 2021-04-04T09:26:41.432 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-04-04T09:26:41.434 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-04-04T09:26:41.472 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:41 +0000 (0:00:00.046) 0:02:38.489 ********** 2021-04-04T09:26:41.481 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-04-04T09:26:41.509 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:41 +0000 (0:00:00.037) 0:02:38.527 ********** 2021-04-04T09:26:41.632 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-04-04T09:26:41.634 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-04-04T09:26:41.676 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-04-04T09:26:41.741 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-04-04T09:26:41.784 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:41 +0000 (0:00:00.274) 0:02:38.801 ********** 2021-04-04T09:26:41.792 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-04-04T09:26:41.823 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:41 +0000 (0:00:00.039) 0:02:38.841 ********** 2021-04-04T09:26:42.156 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-04-04T09:26:42.159 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1617528401.839136-29832-115507236207158/source", "state": "file", "uid": 0} 2021-04-04T09:26:42.178 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1617528401.8802962-29838-267830031326108/source", "state": "file", "uid": 0} 2021-04-04T09:26:42.472 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1617528402.1806328-29832-60109850885466/source", "state": "file", "uid": 0} 2021-04-04T09:26:42.509 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:42 +0000 (0:00:00.685) 0:02:39.527 ********** 2021-04-04T09:26:42.516 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1617528402.2008896-29838-100954669254770/source", "state": "file", "uid": 0} 2021-04-04T09:26:42.550 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:42 +0000 (0:00:00.041) 0:02:39.568 ********** 2021-04-04T09:26:46.632 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-04-04T09:26:46.633 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-L7Gy2g-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-L7Gy2g-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-04-04T09:26:46.635 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-04-04T09:26:46.658 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:46 +0000 (0:00:04.107) 0:02:43.676 ********** 2021-04-04T09:26:46.727 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-rTQBJX-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-rTQBJX-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-04-04T09:26:46.729 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-04-04T09:26:46.753 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:46 +0000 (0:00:00.094) 0:02:43.771 ********** 2021-04-04T09:26:46.791 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-04-04T09:26:46.793 INFO:teuthology.task.ansible.out:ok: [smithi122.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-04-04T09:26:46.822 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:46 +0000 (0:00:00.068) 0:02:43.840 ********** 2021-04-04T09:26:46.891 INFO:teuthology.task.ansible.out:ok: [smithi078.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-04-04T09:26:46.916 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:46 +0000 (0:00:00.094) 0:02:43.934 ********** 2021-04-04T09:26:47.173 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-04-04T09:26:47.174 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1617528406.8713982-30247-139333961835611/source", "state": "file", "uid": 0} 2021-04-04T09:26:47.198 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:47 +0000 (0:00:00.281) 0:02:44.216 ********** 2021-04-04T09:26:47.256 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1617528406.9664469-30256-230691935083132/source", "state": "file", "uid": 0} 2021-04-04T09:26:47.282 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:47 +0000 (0:00:00.083) 0:02:44.299 ********** 2021-04-04T09:26:47.476 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-04-04T09:26:47.478 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.140030", "end": "2021-04-04 09:26:47.782257", "rc": 0, "start": "2021-04-04 09:26:47.642227", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:47.514 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:47 +0000 (0:00:00.231) 0:02:44.532 ********** 2021-04-04T09:26:47.617 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.195732", "end": "2021-04-04 09:26:47.595138", "rc": 0, "start": "2021-04-04 09:26:47.399406", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-04-04T09:26:47.647 INFO:teuthology.task.ansible.out:Sunday 04 April 2021 09:26:47 +0000 (0:00:00.132) 0:02:44.664 ********** 2021-04-04T09:26:50.868 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-04-04T09:26:50.869 INFO:teuthology.task.ansible.out:changed: [smithi122.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "firewalld.service sysinit.target system.slice containerd.service systemd-journald.socket network-online.target basic.target docker.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-04-04T09:26:50.871 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128028", "LimitSIGPENDINGSoft": "12802 2021-04-04T09:26:50.872 INFO:teuthology.task.ansible.out:8", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target docker.socket", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2021-04-04 09:25:42 UTC", "StateChangeTimestampMonotonic": "136897417", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "S 2021-04-04T09:26:50.874 INFO:teuthology.task.ansible.out:uccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-04-04T09:26:51.073 INFO:teuthology.task.ansible.out:changed: [smithi078.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "systemd-journald.socket system.slice containerd.service docker.socket network-online.target sysinit.target basic.target firewalld.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-04-04T09:26:51.075 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128028", "LimitSIGPENDINGSoft": "12802 2021-04-04T09:26:51.076 INFO:teuthology.task.ansible.out:8", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "docker.socket sysinit.target system.slice", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Sun 2021-04-04 09:25:41 UTC", "StateChangeTimestampMonotonic": "144881142", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "S 2021-04-04T09:26:51.078 INFO:teuthology.task.ansible.out:uccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-online.target containerd.service", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-04-04T09:26:51.142 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-04-04T09:26:51.143 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-04-04T09:26:51.144 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi078.front.sepia.ceph.com : ok=127 changed=38 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 smithi122.front.sepia.ceph.com : ok=127 changed=38 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 Sunday 04 April 2021 09:26:51 +0000 (0:00:03.498) 0:02:48.162 ********** =============================================================================== 2021-04-04T09:26:51.145 INFO:teuthology.task.ansible.out:Create all admin users with sudo access. ------------------------------- 21.24s users : Update authorized_keys using the keys repo --------------------- 17.61s testnode : Install packages -------------------------------------------- 10.15s testnode : Ensure packages are not present. ----------------------------- 9.35s testnode : Zap all non-root disks --------------------------------------- 8.02s Remove revoked users ---------------------------------------------------- 5.62s container-host : Install registries-conf-ctl ---------------------------- 4.11s container-host : Restart docker service --------------------------------- 3.50s testnode : Update apt cache. -------------------------------------------- 2.28s Gathering Facts --------------------------------------------------------- 1.77s common : Update apt cache ----------------------------------------------- 1.66s 2021-04-04T09:26:51.145 INFO:teuthology.task.ansible.out:users : Update authorized_keys for each user with literal keys ---------- 1.54s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.53s testnode : Install apt keys --------------------------------------------- 1.45s testnode : Create logical volume(s) ------------------------------------- 1.23s testnode : Make sure apt dependencies are installed --------------------- 0.93s common : Install Kerberos Packages (Debian) ----------------------------- 0.93s testnode : Ensu 2021-04-04T09:26:51.147 INFO:teuthology.task.ansible.out:re perl-doc is installed on apt systems. ----------------- 0.91s container-host : Install container packages ----------------------------- 0.89s testnode : Install packages via pip ------------------------------------- 0.89s 2021-04-04T09:26:51.483 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com')] 2021-04-04T09:26:51.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-04T09:26:51.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-04T09:26:51.566 DEBUG:teuthology.orchestra.run.smithi078:> true 2021-04-04T09:26:51.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-04T09:26:51.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-04-04T09:26:51.829 DEBUG:teuthology.orchestra.run.smithi122:> true 2021-04-04T09:26:52.020 INFO:teuthology.run_tasks:Running task clock... 2021-04-04T09:26:52.042 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-04-04T09:26:52.043 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-04-04T09:26:52.045 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-04-04T09:26:52.103 INFO:teuthology.orchestra.run.smithi078.stdout: 4 Apr 09:26:52 ntpd[8653]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-04-04T09:26:52.103 INFO:teuthology.orchestra.run.smithi078.stdout: 4 Apr 09:26:52 ntpd[8653]: Command line: ntpd -gq 2021-04-04T09:26:52.104 INFO:teuthology.orchestra.run.smithi078.stdout: 4 Apr 09:26:52 ntpd[8653]: proto: precision = 0.111 usec (-23) 2021-04-04T09:26:52.104 INFO:teuthology.orchestra.run.smithi078.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-04-04T09:26:52.105 INFO:teuthology.orchestra.run.smithi078.stdout: 4 Apr 09:26:52 ntpd[8653]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-04-04T09:26:52.105 INFO:teuthology.orchestra.run.smithi078.stdout: 4 Apr 09:26:52 ntpd[8653]: restrict ::: KOD does nothing without LIMITED. 2021-04-04T09:26:52.105 INFO:teuthology.orchestra.run.smithi078.stderr:restrict ::: KOD does nothing without LIMITED. 2021-04-04T09:26:52.106 INFO:teuthology.orchestra.run.smithi078.stdout: 4 Apr 09:26:52 ntpd[8653]: Listen and drop on 0 v6wildcard [::]:123 2021-04-04T09:26:52.107 INFO:teuthology.orchestra.run.smithi078.stdout: 4 Apr 09:26:52 ntpd[8653]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-04-04T09:26:52.107 INFO:teuthology.orchestra.run.smithi078.stdout: 4 Apr 09:26:52 ntpd[8653]: Listen normally on 2 lo 127.0.0.1:123 2021-04-04T09:26:52.107 INFO:teuthology.orchestra.run.smithi078.stdout: 4 Apr 09:26:52 ntpd[8653]: Listen normally on 3 enp3s0f1 172.21.15.78:123 2021-04-04T09:26:52.107 INFO:teuthology.orchestra.run.smithi078.stdout: 4 Apr 09:26:52 ntpd[8653]: Listen normally on 4 lo [::1]:123 2021-04-04T09:26:52.108 INFO:teuthology.orchestra.run.smithi078.stdout: 4 Apr 09:26:52 ntpd[8653]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d4dd%5]:123 2021-04-04T09:26:52.108 INFO:teuthology.orchestra.run.smithi078.stdout: 4 Apr 09:26:52 ntpd[8653]: Listening on routing socket on fd #22 for interface updates 2021-04-04T09:26:53.104 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:53 ntpd[8653]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:26:53.104 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:53 ntpd[8653]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:26:53.104 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:53 ntpd[8653]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:26:53.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:53 ntpd[8653]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:26:53.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:53 ntpd[8653]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:26:53.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:53 ntpd[8653]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:26:53.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:53 ntpd[8653]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:26:53.106 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:53 ntpd[8653]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:26:55.104 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:55 ntpd[8653]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:26:55.104 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:55 ntpd[8653]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:26:55.104 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:55 ntpd[8653]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:26:55.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:55 ntpd[8653]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:26:55.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:55 ntpd[8653]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:26:55.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:55 ntpd[8653]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:26:55.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:55 ntpd[8653]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:26:55.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:55 ntpd[8653]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:26:56.104 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:56 ntpd[8653]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:26:56.104 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:56 ntpd[8653]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:26:56.104 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:56 ntpd[8653]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:26:56.104 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:56 ntpd[8653]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:26:57.104 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:57 ntpd[8653]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:26:57.104 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:57 ntpd[8653]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:26:57.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:57 ntpd[8653]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:26:57.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:57 ntpd[8653]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:26:57.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:57 ntpd[8653]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:26:57.106 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:57 ntpd[8653]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:26:57.106 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:57 ntpd[8653]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:26:57.106 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:57 ntpd[8653]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:26:57.106 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:57 ntpd[8653]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:26:57.106 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:57 ntpd[8653]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:26:57.107 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:57 ntpd[8653]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:26:57.107 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:57 ntpd[8653]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:26:58.103 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:58 ntpd[8653]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:26:58.104 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:58 ntpd[8653]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:26:58.104 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:58 ntpd[8653]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:26:58.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:58 ntpd[8653]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:26:59.104 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:59 ntpd[8653]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:26:59.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:59 ntpd[8653]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:26:59.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:59 ntpd[8653]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:26:59.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:59 ntpd[8653]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:26:59.105 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:59 ntpd[8653]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:26:59.106 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:59 ntpd[8653]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:26:59.106 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:59 ntpd[8653]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:26:59.106 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:59 ntpd[8653]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:26:59.106 INFO:teuthology.orchestra.run.smithi078.stdout: 4 Apr 09:26:59 ntpd[8653]: ntpd: time slew -0.003337 s 2021-04-04T09:26:59.107 INFO:teuthology.orchestra.run.smithi078.stdout:ntpd: time slew -0.003337s 2021-04-04T09:26:59.107 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:59 ntpd[8653]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-04-04T09:26:59.108 INFO:teuthology.orchestra.run.smithi078.stderr: 4 Apr 09:26:59 ntpd[8653]: can't open /var/log/ntpstats/loopstats.20210404: Permission denied 2021-04-04T09:26:59.167 INFO:teuthology.orchestra.run.smithi078.stdout: remote refid st t when poll reach delay offset jitter 2021-04-04T09:26:59.167 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================== 2021-04-04T09:26:59.167 INFO:teuthology.orchestra.run.smithi078.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-04T09:26:59.167 INFO:teuthology.orchestra.run.smithi078.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-04T09:26:59.168 INFO:teuthology.orchestra.run.smithi078.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-04T09:26:59.168 INFO:teuthology.orchestra.run.smithi078.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-04T09:26:59.168 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-04-04T09:26:59.170 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-04-04T09:26:59.223 INFO:teuthology.orchestra.run.smithi122.stdout: 4 Apr 09:26:59 ntpd[8533]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-04-04T09:26:59.224 INFO:teuthology.orchestra.run.smithi122.stdout: 4 Apr 09:26:59 ntpd[8533]: Command line: ntpd -gq 2021-04-04T09:26:59.224 INFO:teuthology.orchestra.run.smithi122.stdout: 4 Apr 09:26:59 ntpd[8533]: proto: precision = 0.103 usec (-23) 2021-04-04T09:26:59.225 INFO:teuthology.orchestra.run.smithi122.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-04-04T09:26:59.226 INFO:teuthology.orchestra.run.smithi122.stdout: 4 Apr 09:26:59 ntpd[8533]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-04-04T09:26:59.226 INFO:teuthology.orchestra.run.smithi122.stderr:restrict ::: KOD does nothing without LIMITED. 2021-04-04T09:26:59.227 INFO:teuthology.orchestra.run.smithi122.stdout: 4 Apr 09:26:59 ntpd[8533]: restrict ::: KOD does nothing without LIMITED. 2021-04-04T09:26:59.228 INFO:teuthology.orchestra.run.smithi122.stdout: 4 Apr 09:26:59 ntpd[8533]: Listen and drop on 0 v6wildcard [::]:123 2021-04-04T09:26:59.228 INFO:teuthology.orchestra.run.smithi122.stdout: 4 Apr 09:26:59 ntpd[8533]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-04-04T09:26:59.228 INFO:teuthology.orchestra.run.smithi122.stdout: 4 Apr 09:26:59 ntpd[8533]: Listen normally on 2 lo 127.0.0.1:123 2021-04-04T09:26:59.228 INFO:teuthology.orchestra.run.smithi122.stdout: 4 Apr 09:26:59 ntpd[8533]: Listen normally on 3 enp3s0f1 172.21.15.122:123 2021-04-04T09:26:59.229 INFO:teuthology.orchestra.run.smithi122.stdout: 4 Apr 09:26:59 ntpd[8533]: Listen normally on 4 lo [::1]:123 2021-04-04T09:26:59.229 INFO:teuthology.orchestra.run.smithi122.stdout: 4 Apr 09:26:59 ntpd[8533]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d515%5]:123 2021-04-04T09:26:59.229 INFO:teuthology.orchestra.run.smithi122.stdout: 4 Apr 09:26:59 ntpd[8533]: Listening on routing socket on fd #22 for interface updates 2021-04-04T09:27:00.224 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:00 ntpd[8533]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:27:00.225 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:00 ntpd[8533]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:27:00.225 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:00 ntpd[8533]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:27:00.226 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:00 ntpd[8533]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:27:00.226 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:00 ntpd[8533]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:27:00.226 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:00 ntpd[8533]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:27:00.226 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:00 ntpd[8533]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:27:00.227 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:00 ntpd[8533]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:27:02.226 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:02 ntpd[8533]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:27:02.226 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:02 ntpd[8533]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:27:02.226 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:02 ntpd[8533]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:27:02.227 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:02 ntpd[8533]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:27:02.227 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:02 ntpd[8533]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:27:02.227 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:02 ntpd[8533]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:27:02.227 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:02 ntpd[8533]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:27:02.228 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:02 ntpd[8533]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:27:02.228 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:02 ntpd[8533]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:27:02.228 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:02 ntpd[8533]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:27:02.229 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:02 ntpd[8533]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:27:02.229 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:02 ntpd[8533]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:27:03.225 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:03 ntpd[8533]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:27:03.226 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:03 ntpd[8533]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:27:03.226 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:03 ntpd[8533]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:27:03.227 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:03 ntpd[8533]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:27:04.227 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:04 ntpd[8533]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:27:04.227 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:04 ntpd[8533]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:27:04.227 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:04 ntpd[8533]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:27:04.228 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:04 ntpd[8533]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:27:04.228 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:04 ntpd[8533]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:27:04.228 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:04 ntpd[8533]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:27:04.228 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:04 ntpd[8533]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:27:04.229 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:04 ntpd[8533]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:27:04.229 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:04 ntpd[8533]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:27:04.229 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:04 ntpd[8533]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:27:04.229 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:04 ntpd[8533]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:27:04.230 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:04 ntpd[8533]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:27:05.226 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:05 ntpd[8533]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:27:05.227 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:05 ntpd[8533]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:27:05.227 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:05 ntpd[8533]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:27:05.227 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:05 ntpd[8533]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:27:06.227 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:06 ntpd[8533]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-04-04T09:27:06.227 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:06 ntpd[8533]: can't open /var/log/ntpstats/rawstats.20210404: Permission denied 2021-04-04T09:27:06.227 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:06 ntpd[8533]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-04-04T09:27:06.228 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:06 ntpd[8533]: can't open /var/log/ntpstats/peerstats.20210404: Permission denied 2021-04-04T09:27:06.228 INFO:teuthology.orchestra.run.smithi122.stdout: 4 Apr 09:27:06 ntpd[8533]: ntpd: time set -0.315372 s 2021-04-04T09:27:06.228 INFO:teuthology.orchestra.run.smithi122.stdout:ntpd: time set -0.315372s 2021-04-04T09:27:06.229 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:06 ntpd[8533]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-04-04T09:27:06.229 INFO:teuthology.orchestra.run.smithi122.stderr: 4 Apr 09:27:06 ntpd[8533]: can't open /var/log/ntpstats/loopstats.20210404: Permission denied 2021-04-04T09:27:06.282 INFO:teuthology.orchestra.run.smithi122.stdout: remote refid st t when poll reach delay offset jitter 2021-04-04T09:27:06.282 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================== 2021-04-04T09:27:06.283 INFO:teuthology.orchestra.run.smithi122.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-04T09:27:06.283 INFO:teuthology.orchestra.run.smithi122.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-04T09:27:06.283 INFO:teuthology.orchestra.run.smithi122.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-04T09:27:06.284 INFO:teuthology.orchestra.run.smithi122.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-04-04T09:27:06.285 INFO:teuthology.run_tasks:Running task install... 2021-04-04T09:27:06.294 DEBUG:teuthology.task.install:project ceph 2021-04-04T09:27:06.296 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b'}} 2021-04-04T09:27:06.297 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b'} 2021-04-04T09:27:06.298 INFO:teuthology.task.install:Using flavor: basic 2021-04-04T09:27:06.312 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-04-04T09:27:06.313 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2021-04-04T09:27:06.313 DEBUG:teuthology.orchestra.run.smithi078:> sudo lsb_release -is 2021-04-04T09:27:06.366 INFO:teuthology.orchestra.run.smithi078.stdout:Ubuntu 2021-04-04T09:27:06.371 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-04-04T09:27:06.372 DEBUG:teuthology.orchestra.run.smithi122:> sudo lsb_release -is 2021-04-04T09:27:06.372 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-key list | grep Ceph 2021-04-04T09:27:06.424 INFO:teuthology.orchestra.run.smithi122.stdout:Ubuntu 2021-04-04T09:27:06.430 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-04-04T09:27:06.430 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-key list | grep Ceph 2021-04-04T09:27:06.464 INFO:teuthology.orchestra.run.smithi078.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-04-04T09:27:06.517 INFO:teuthology.orchestra.run.smithi122.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-04-04T09:27:06.518 INFO:teuthology.orchestra.run.smithi078.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-04-04T09:27:06.519 INFO:teuthology.orchestra.run.smithi078.stdout:uid [ unknown] Ceph.com (release key) 2021-04-04T09:27:06.520 DEBUG:teuthology.orchestra.run.smithi078:> sudo lsb_release -is 2021-04-04T09:27:06.570 INFO:teuthology.orchestra.run.smithi078.stdout:Ubuntu 2021-04-04T09:27:06.574 INFO:teuthology.orchestra.run.smithi122.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-04-04T09:27:06.575 INFO:teuthology.orchestra.run.smithi122.stdout:uid [ unknown] Ceph.com (release key) 2021-04-04T09:27:06.576 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-04-04T09:27:06.577 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2021-04-04T09:27:06.578 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=f4fb41419a42bee43a2069637ef17064073bf30b 2021-04-04T09:27:06.582 DEBUG:teuthology.orchestra.run.smithi122:> sudo lsb_release -is 2021-04-04T09:27:06.638 INFO:teuthology.orchestra.run.smithi122.stdout:Ubuntu 2021-04-04T09:27:06.643 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-04-04T09:27:06.643 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2021-04-04T09:27:06.644 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=f4fb41419a42bee43a2069637ef17064073bf30b 2021-04-04T09:27:06.724 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default/ 2021-04-04T09:27:06.725 INFO:teuthology.task.install.deb:Package version is 16.2.0-22-gf4fb4141-1bionic 2021-04-04T09:27:06.791 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default/ 2021-04-04T09:27:06.792 INFO:teuthology.task.install.deb:Package version is 16.2.0-22-gf4fb4141-1bionic 2021-04-04T09:27:06.843 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:27:06.843 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-04-04T09:27:06.859 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get update 2021-04-04T09:27:06.900 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:27:06.900 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-04-04T09:27:06.914 DEBUG:teuthology.orchestra.run.smithi122:> sudo apt-get update 2021-04-04T09:27:07.096 INFO:teuthology.orchestra.run.smithi078.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic InRelease 2021-04-04T09:27:07.117 INFO:teuthology.orchestra.run.smithi078.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-04-04T09:27:07.140 INFO:teuthology.orchestra.run.smithi078.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic Release.gpg 2021-04-04T09:27:07.145 INFO:teuthology.orchestra.run.smithi122.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic InRelease 2021-04-04T09:27:07.166 INFO:teuthology.orchestra.run.smithi122.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-04-04T09:27:07.167 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-04-04T09:27:07.167 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-04-04T09:27:07.168 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,113 B] 2021-04-04T09:27:07.189 INFO:teuthology.orchestra.run.smithi122.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic Release.gpg 2021-04-04T09:27:07.202 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 Packages [17.0 kB] 2021-04-04T09:27:07.218 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-04-04T09:27:07.219 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-04-04T09:27:07.219 INFO:teuthology.orchestra.run.smithi122.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 Packages [17.0 kB] 2021-04-04T09:27:07.237 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-04-04T09:27:07.260 INFO:teuthology.orchestra.run.smithi122.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,113 B] 2021-04-04T09:27:07.285 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-04-04T09:27:07.317 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-04-04T09:27:07.365 INFO:teuthology.orchestra.run.smithi122.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-04-04T09:27:07.759 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 27.8 kB in 1s (39.7 kB/s) 2021-04-04T09:27:07.828 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 27.8 kB in 1s (38.7 kB/s) 2021-04-04T09:27:09.001 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2021-04-04T09:27:09.047 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.2.0-22-gf4fb4141-1bionic cephadm=16.2.0-22-gf4fb4141-1bionic ceph-mds=16.2.0-22-gf4fb4141-1bionic ceph-mgr=16.2.0-22-gf4fb4141-1bionic ceph-common=16.2.0-22-gf4fb4141-1bionic ceph-fuse=16.2.0-22-gf4fb4141-1bionic ceph-test=16.2.0-22-gf4fb4141-1bionic radosgw=16.2.0-22-gf4fb4141-1bionic python3-rados=16.2.0-22-gf4fb4141-1bionic python3-rgw=16.2.0-22-gf4fb4141-1bionic python3-cephfs=16.2.0-22-gf4fb4141-1bionic python3-rbd=16.2.0-22-gf4fb4141-1bionic libcephfs2=16.2.0-22-gf4fb4141-1bionic libcephfs-dev=16.2.0-22-gf4fb4141-1bionic librados2=16.2.0-22-gf4fb4141-1bionic librbd1=16.2.0-22-gf4fb4141-1bionic rbd-fuse=16.2.0-22-gf4fb4141-1bionic rbd-mirror=16.2.0-22-gf4fb4141-1bionic 2021-04-04T09:27:09.086 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2021-04-04T09:27:09.123 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2021-04-04T09:27:09.132 DEBUG:teuthology.orchestra.run.smithi122:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.2.0-22-gf4fb4141-1bionic cephadm=16.2.0-22-gf4fb4141-1bionic ceph-mds=16.2.0-22-gf4fb4141-1bionic ceph-mgr=16.2.0-22-gf4fb4141-1bionic ceph-common=16.2.0-22-gf4fb4141-1bionic ceph-fuse=16.2.0-22-gf4fb4141-1bionic ceph-test=16.2.0-22-gf4fb4141-1bionic radosgw=16.2.0-22-gf4fb4141-1bionic python3-rados=16.2.0-22-gf4fb4141-1bionic python3-rgw=16.2.0-22-gf4fb4141-1bionic python3-cephfs=16.2.0-22-gf4fb4141-1bionic python3-rbd=16.2.0-22-gf4fb4141-1bionic libcephfs2=16.2.0-22-gf4fb4141-1bionic libcephfs-dev=16.2.0-22-gf4fb4141-1bionic librados2=16.2.0-22-gf4fb4141-1bionic librbd1=16.2.0-22-gf4fb4141-1bionic rbd-fuse=16.2.0-22-gf4fb4141-1bionic rbd-mirror=16.2.0-22-gf4fb4141-1bionic 2021-04-04T09:27:09.209 INFO:teuthology.orchestra.run.smithi122.stdout:Reading package lists... 2021-04-04T09:27:09.384 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2021-04-04T09:27:09.385 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2021-04-04T09:27:09.448 INFO:teuthology.orchestra.run.smithi122.stdout:Building dependency tree... 2021-04-04T09:27:09.449 INFO:teuthology.orchestra.run.smithi122.stdout:Reading state information... 2021-04-04T09:27:09.556 INFO:teuthology.orchestra.run.smithi078.stdout:The following additional packages will be installed: 2021-04-04T09:27:09.556 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-04-04T09:27:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-04-04T09:27:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: libjaeger libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-04-04T09:27:09.557 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libsqlite3-mod-ceph 2021-04-04T09:27:09.558 INFO:teuthology.orchestra.run.smithi078.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2021-04-04T09:27:09.558 INFO:teuthology.orchestra.run.smithi078.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2021-04-04T09:27:09.558 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2021-04-04T09:27:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-04-04T09:27:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-04-04T09:27:09.559 INFO:teuthology.orchestra.run.smithi078.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-04-04T09:27:09.560 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-04-04T09:27:09.560 INFO:teuthology.orchestra.run.smithi078.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-04-04T09:27:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-04-04T09:27:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-04-04T09:27:09.561 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-04-04T09:27:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-04-04T09:27:09.562 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytest python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-04-04T09:27:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-04-04T09:27:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-04-04T09:27:09.563 INFO:teuthology.orchestra.run.smithi078.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-04-04T09:27:09.564 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-04-04T09:27:09.564 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-04-04T09:27:09.565 INFO:teuthology.orchestra.run.smithi078.stdout:Suggested packages: 2021-04-04T09:27:09.565 INFO:teuthology.orchestra.run.smithi078.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-04-04T09:27:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-04-04T09:27:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-04-04T09:27:09.566 INFO:teuthology.orchestra.run.smithi078.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-04-04T09:27:09.567 INFO:teuthology.orchestra.run.smithi078.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-04-04T09:27:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-04-04T09:27:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-04-04T09:27:09.568 INFO:teuthology.orchestra.run.smithi078.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-04-04T09:27:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-04-04T09:27:09.569 INFO:teuthology.orchestra.run.smithi078.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-04-04T09:27:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-04-04T09:27:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-04-04T09:27:09.570 INFO:teuthology.orchestra.run.smithi078.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-04-04T09:27:09.571 INFO:teuthology.orchestra.run.smithi078.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-04-04T09:27:09.571 INFO:teuthology.orchestra.run.smithi078.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-04-04T09:27:09.616 INFO:teuthology.orchestra.run.smithi122.stdout:The following additional packages will be installed: 2021-04-04T09:27:09.616 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2021-04-04T09:27:09.618 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd jq 2021-04-04T09:27:09.619 INFO:teuthology.orchestra.run.smithi122.stdout: libjaeger libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-04-04T09:27:09.619 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libsqlite3-mod-ceph 2021-04-04T09:27:09.620 INFO:teuthology.orchestra.run.smithi122.stdout: libxslt1.1 nvme-cli python-attr python-funcsigs python-pastedeploy-tpl 2021-04-04T09:27:09.620 INFO:teuthology.orchestra.run.smithi122.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2021-04-04T09:27:09.621 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2021-04-04T09:27:09.621 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-04-04T09:27:09.621 INFO:teuthology.orchestra.run.smithi122.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-04-04T09:27:09.622 INFO:teuthology.orchestra.run.smithi122.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-04-04T09:27:09.622 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-04-04T09:27:09.623 INFO:teuthology.orchestra.run.smithi122.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-04-04T09:27:09.624 INFO:teuthology.orchestra.run.smithi122.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-04-04T09:27:09.624 INFO:teuthology.orchestra.run.smithi122.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-04-04T09:27:09.624 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-04-04T09:27:09.625 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-04-04T09:27:09.625 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytest python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-04-04T09:27:09.626 INFO:teuthology.orchestra.run.smithi122.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-04-04T09:27:09.626 INFO:teuthology.orchestra.run.smithi122.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-04-04T09:27:09.626 INFO:teuthology.orchestra.run.smithi122.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-04-04T09:27:09.627 INFO:teuthology.orchestra.run.smithi122.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-04-04T09:27:09.627 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-04-04T09:27:09.628 INFO:teuthology.orchestra.run.smithi122.stdout:Suggested packages: 2021-04-04T09:27:09.629 INFO:teuthology.orchestra.run.smithi122.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-04-04T09:27:09.629 INFO:teuthology.orchestra.run.smithi122.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-04-04T09:27:09.630 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-04-04T09:27:09.630 INFO:teuthology.orchestra.run.smithi122.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-04-04T09:27:09.631 INFO:teuthology.orchestra.run.smithi122.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-04-04T09:27:09.631 INFO:teuthology.orchestra.run.smithi122.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-04-04T09:27:09.631 INFO:teuthology.orchestra.run.smithi122.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-04-04T09:27:09.632 INFO:teuthology.orchestra.run.smithi122.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-04-04T09:27:09.632 INFO:teuthology.orchestra.run.smithi122.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-04-04T09:27:09.633 INFO:teuthology.orchestra.run.smithi122.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-04-04T09:27:09.633 INFO:teuthology.orchestra.run.smithi122.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-04-04T09:27:09.633 INFO:teuthology.orchestra.run.smithi122.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-04-04T09:27:09.634 INFO:teuthology.orchestra.run.smithi122.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-04-04T09:27:09.634 INFO:teuthology.orchestra.run.smithi122.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-04-04T09:27:09.634 INFO:teuthology.orchestra.run.smithi122.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-04-04T09:27:09.661 INFO:teuthology.orchestra.run.smithi078.stdout:The following NEW packages will be installed: 2021-04-04T09:27:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-04-04T09:27:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-04-04T09:27:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-04-04T09:27:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libleveldb1v5 liblttng-ust0 2021-04-04T09:27:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-04-04T09:27:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: libsnappy1v5 libsqlite3-mod-ceph libxslt1.1 nvme-cli python-attr 2021-04-04T09:27:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: python-funcsigs python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-04-04T09:27:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-04-04T09:27:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-04-04T09:27:09.667 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-04-04T09:27:09.667 INFO:teuthology.orchestra.run.smithi078.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-04-04T09:27:09.667 INFO:teuthology.orchestra.run.smithi078.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-04-04T09:27:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-04-04T09:27:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-04-04T09:27:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-04-04T09:27:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-04-04T09:27:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-04-04T09:27:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-04-04T09:27:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-04-04T09:27:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-04-04T09:27:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-04-04T09:27:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-04-04T09:27:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-04-04T09:27:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-04-04T09:27:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug radosgw rbd-fuse rbd-mirror socat xmlstarlet 2021-04-04T09:27:09.671 INFO:teuthology.orchestra.run.smithi078.stdout:The following packages will be upgraded: 2021-04-04T09:27:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: librados2 librbd1 2021-04-04T09:27:09.724 INFO:teuthology.orchestra.run.smithi122.stdout:The following NEW packages will be installed: 2021-04-04T09:27:09.725 INFO:teuthology.orchestra.run.smithi122.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-04-04T09:27:09.725 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2021-04-04T09:27:09.727 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq 2021-04-04T09:27:09.727 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-dev libcephfs2 libjaeger libjq1 libleveldb1v5 liblttng-ust0 2021-04-04T09:27:09.727 INFO:teuthology.orchestra.run.smithi122.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-04-04T09:27:09.727 INFO:teuthology.orchestra.run.smithi122.stdout: libsnappy1v5 libsqlite3-mod-ceph libxslt1.1 nvme-cli python-attr 2021-04-04T09:27:09.728 INFO:teuthology.orchestra.run.smithi122.stdout: python-funcsigs python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-04-04T09:27:09.728 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-04-04T09:27:09.728 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-04-04T09:27:09.728 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-04-04T09:27:09.729 INFO:teuthology.orchestra.run.smithi122.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-04-04T09:27:09.729 INFO:teuthology.orchestra.run.smithi122.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-04-04T09:27:09.729 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-04-04T09:27:09.729 INFO:teuthology.orchestra.run.smithi122.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-04-04T09:27:09.729 INFO:teuthology.orchestra.run.smithi122.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-04-04T09:27:09.730 INFO:teuthology.orchestra.run.smithi122.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-04-04T09:27:09.730 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-04-04T09:27:09.730 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-04-04T09:27:09.730 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-04-04T09:27:09.730 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-04-04T09:27:09.731 INFO:teuthology.orchestra.run.smithi122.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-04-04T09:27:09.731 INFO:teuthology.orchestra.run.smithi122.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-04-04T09:27:09.731 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-04-04T09:27:09.732 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-04-04T09:27:09.732 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug radosgw rbd-fuse rbd-mirror socat xmlstarlet 2021-04-04T09:27:09.733 INFO:teuthology.orchestra.run.smithi122.stdout:The following packages will be upgraded: 2021-04-04T09:27:09.733 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 librbd1 2021-04-04T09:27:09.791 INFO:teuthology.orchestra.run.smithi078.stdout:2 upgraded, 113 newly installed, 0 to remove and 35 not upgraded. 2021-04-04T09:27:09.791 INFO:teuthology.orchestra.run.smithi078.stdout:Need to get 178 MB of archives. 2021-04-04T09:27:09.792 INFO:teuthology.orchestra.run.smithi078.stdout:After this operation, 862 MB of additional disk space will be used. 2021-04-04T09:27:09.792 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 libjaeger amd64 16.2.0-22-gf4fb4141-1bionic [3,748 B] 2021-04-04T09:27:09.832 INFO:teuthology.orchestra.run.smithi078.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.2.0-22-gf4fb4141-1bionic [3,155 kB] 2021-04-04T09:27:09.852 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-04-04T09:27:09.856 INFO:teuthology.orchestra.run.smithi122.stdout:2 upgraded, 113 newly installed, 0 to remove and 35 not upgraded. 2021-04-04T09:27:09.857 INFO:teuthology.orchestra.run.smithi122.stdout:Need to get 178 MB of archives. 2021-04-04T09:27:09.857 INFO:teuthology.orchestra.run.smithi122.stdout:After this operation, 862 MB of additional disk space will be used. 2021-04-04T09:27:09.857 INFO:teuthology.orchestra.run.smithi122.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 libjaeger amd64 16.2.0-22-gf4fb4141-1bionic [3,748 B] 2021-04-04T09:27:09.898 INFO:teuthology.orchestra.run.smithi122.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 16.2.0-22-gf4fb4141-1bionic [3,155 kB] 2021-04-04T09:27:09.919 INFO:teuthology.orchestra.run.smithi122.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-04-04T09:27:09.933 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-04-04T09:27:10.000 INFO:teuthology.orchestra.run.smithi122.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-04-04T09:27:10.172 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.2.0-22-gf4fb4141-1bionic [3,505 kB] 2021-04-04T09:27:10.190 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-04-04T09:27:10.199 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-04-04T09:27:10.207 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-04-04T09:27:10.241 INFO:teuthology.orchestra.run.smithi122.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 16.2.0-22-gf4fb4141-1bionic [3,505 kB] 2021-04-04T09:27:10.257 INFO:teuthology.orchestra.run.smithi122.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-04-04T09:27:10.264 INFO:teuthology.orchestra.run.smithi122.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-04-04T09:27:10.280 INFO:teuthology.orchestra.run.smithi122.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-04-04T09:27:10.312 INFO:teuthology.orchestra.run.smithi078.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-04-04T09:27:10.316 INFO:teuthology.orchestra.run.smithi078.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-04-04T09:27:10.349 INFO:teuthology.orchestra.run.smithi078.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-04-04T09:27:10.359 INFO:teuthology.orchestra.run.smithi078.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-04-04T09:27:10.368 INFO:teuthology.orchestra.run.smithi078.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-04-04T09:27:10.373 INFO:teuthology.orchestra.run.smithi078.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.2.0-22-gf4fb4141-1bionic [676 kB] 2021-04-04T09:27:10.374 INFO:teuthology.orchestra.run.smithi122.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-04-04T09:27:10.377 INFO:teuthology.orchestra.run.smithi122.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-04-04T09:27:10.378 INFO:teuthology.orchestra.run.smithi078.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-04-04T09:27:10.387 INFO:teuthology.orchestra.run.smithi078.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-04-04T09:27:10.411 INFO:teuthology.orchestra.run.smithi078.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.2.0-22-gf4fb4141-1bionic [339 kB] 2021-04-04T09:27:10.416 INFO:teuthology.orchestra.run.smithi078.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-04-04T09:27:10.418 INFO:teuthology.orchestra.run.smithi122.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-04-04T09:27:10.420 INFO:teuthology.orchestra.run.smithi078.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-04-04T09:27:10.426 INFO:teuthology.orchestra.run.smithi122.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-04-04T09:27:10.431 INFO:teuthology.orchestra.run.smithi078.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-04-04T09:27:10.432 INFO:teuthology.orchestra.run.smithi122.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-04-04T09:27:10.433 INFO:teuthology.orchestra.run.smithi078.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.2.0-22-gf4fb4141-1bionic [21.7 kB] 2021-04-04T09:27:10.436 INFO:teuthology.orchestra.run.smithi078.stdout:Get:22 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.2.0-22-gf4fb4141-1bionic [176 kB] 2021-04-04T09:27:10.442 INFO:teuthology.orchestra.run.smithi122.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-04-04T09:27:10.442 INFO:teuthology.orchestra.run.smithi078.stdout:Get:23 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.2.0-22-gf4fb4141-1bionic [30.0 kB] 2021-04-04T09:27:10.443 INFO:teuthology.orchestra.run.smithi078.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-04-04T09:27:10.444 INFO:teuthology.orchestra.run.smithi078.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-04-04T09:27:10.450 INFO:teuthology.orchestra.run.smithi122.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-04-04T09:27:10.452 INFO:teuthology.orchestra.run.smithi078.stdout:Get:26 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.2.0-22-gf4fb4141-1bionic [335 kB] 2021-04-04T09:27:10.467 INFO:teuthology.orchestra.run.smithi078.stdout:Get:27 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.2.0-22-gf4fb4141-1bionic [3,307 kB] 2021-04-04T09:27:10.483 INFO:teuthology.orchestra.run.smithi122.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 16.2.0-22-gf4fb4141-1bionic [676 kB] 2021-04-04T09:27:10.484 INFO:teuthology.orchestra.run.smithi122.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-04-04T09:27:10.487 INFO:teuthology.orchestra.run.smithi122.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-04-04T09:27:10.498 INFO:teuthology.orchestra.run.smithi122.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-04-04T09:27:10.510 INFO:teuthology.orchestra.run.smithi122.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-04-04T09:27:10.511 INFO:teuthology.orchestra.run.smithi078.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-04-04T09:27:10.512 INFO:teuthology.orchestra.run.smithi078.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-04-04T09:27:10.514 INFO:teuthology.orchestra.run.smithi078.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-04-04T09:27:10.515 INFO:teuthology.orchestra.run.smithi078.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-04-04T09:27:10.515 INFO:teuthology.orchestra.run.smithi078.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-04-04T09:27:10.520 INFO:teuthology.orchestra.run.smithi078.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-04-04T09:27:10.521 INFO:teuthology.orchestra.run.smithi078.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-04-04T09:27:10.534 INFO:teuthology.orchestra.run.smithi122.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 16.2.0-22-gf4fb4141-1bionic [339 kB] 2021-04-04T09:27:10.560 INFO:teuthology.orchestra.run.smithi122.stdout:Get:22 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 16.2.0-22-gf4fb4141-1bionic [21.7 kB] 2021-04-04T09:27:10.560 INFO:teuthology.orchestra.run.smithi122.stdout:Get:23 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 16.2.0-22-gf4fb4141-1bionic [176 kB] 2021-04-04T09:27:10.577 INFO:teuthology.orchestra.run.smithi122.stdout:Get:24 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 16.2.0-22-gf4fb4141-1bionic [30.0 kB] 2021-04-04T09:27:10.578 INFO:teuthology.orchestra.run.smithi122.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-04-04T09:27:10.579 INFO:teuthology.orchestra.run.smithi122.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-04-04T09:27:10.579 INFO:teuthology.orchestra.run.smithi122.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-04-04T09:27:10.580 INFO:teuthology.orchestra.run.smithi122.stdout:Get:28 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 16.2.0-22-gf4fb4141-1bionic [335 kB] 2021-04-04T09:27:10.582 INFO:teuthology.orchestra.run.smithi122.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-04-04T09:27:10.582 INFO:teuthology.orchestra.run.smithi122.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-04-04T09:27:10.583 INFO:teuthology.orchestra.run.smithi122.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-04-04T09:27:10.585 INFO:teuthology.orchestra.run.smithi122.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-04-04T09:27:10.587 INFO:teuthology.orchestra.run.smithi122.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-04-04T09:27:10.591 INFO:teuthology.orchestra.run.smithi078.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-04-04T09:27:10.592 INFO:teuthology.orchestra.run.smithi078.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-04-04T09:27:10.602 INFO:teuthology.orchestra.run.smithi122.stdout:Get:34 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 16.2.0-22-gf4fb4141-1bionic [3,307 kB] 2021-04-04T09:27:10.612 INFO:teuthology.orchestra.run.smithi078.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-04-04T09:27:10.659 INFO:teuthology.orchestra.run.smithi122.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-04-04T09:27:10.659 INFO:teuthology.orchestra.run.smithi122.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-04-04T09:27:10.671 INFO:teuthology.orchestra.run.smithi078.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-04-04T09:27:10.672 INFO:teuthology.orchestra.run.smithi078.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-04-04T09:27:10.674 INFO:teuthology.orchestra.run.smithi078.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-04-04T09:27:10.675 INFO:teuthology.orchestra.run.smithi078.stdout:Get:41 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.2.0-22-gf4fb4141-1bionic [99.6 kB] 2021-04-04T09:27:10.679 INFO:teuthology.orchestra.run.smithi078.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-04-04T09:27:10.680 INFO:teuthology.orchestra.run.smithi078.stdout:Get:43 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.2.0-22-gf4fb4141-1bionic [427 kB] 2021-04-04T09:27:10.684 INFO:teuthology.orchestra.run.smithi078.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-04-04T09:27:10.687 INFO:teuthology.orchestra.run.smithi078.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-04-04T09:27:10.687 INFO:teuthology.orchestra.run.smithi078.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-04-04T09:27:10.696 INFO:teuthology.orchestra.run.smithi078.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.2.0-22-gf4fb4141-1bionic [22.0 MB] 2021-04-04T09:27:10.739 INFO:teuthology.orchestra.run.smithi122.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-04-04T09:27:10.739 INFO:teuthology.orchestra.run.smithi122.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-04-04T09:27:10.740 INFO:teuthology.orchestra.run.smithi122.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-04-04T09:27:10.741 INFO:teuthology.orchestra.run.smithi122.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-04-04T09:27:10.751 INFO:teuthology.orchestra.run.smithi078.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-04-04T09:27:10.752 INFO:teuthology.orchestra.run.smithi122.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-04-04T09:27:10.753 INFO:teuthology.orchestra.run.smithi122.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-04-04T09:27:10.759 INFO:teuthology.orchestra.run.smithi122.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-04-04T09:27:10.760 INFO:teuthology.orchestra.run.smithi122.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-04-04T09:27:10.818 INFO:teuthology.orchestra.run.smithi122.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-04-04T09:27:10.837 INFO:teuthology.orchestra.run.smithi078.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-04-04T09:27:10.838 INFO:teuthology.orchestra.run.smithi078.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-04-04T09:27:10.843 INFO:teuthology.orchestra.run.smithi122.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 16.2.0-22-gf4fb4141-1bionic [99.6 kB] 2021-04-04T09:27:10.846 INFO:teuthology.orchestra.run.smithi122.stdout:Get:47 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 16.2.0-22-gf4fb4141-1bionic [427 kB] 2021-04-04T09:27:10.876 INFO:teuthology.orchestra.run.smithi122.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 16.2.0-22-gf4fb4141-1bionic [22.0 MB] 2021-04-04T09:27:10.908 INFO:teuthology.orchestra.run.smithi122.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-04-04T09:27:10.908 INFO:teuthology.orchestra.run.smithi122.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-04-04T09:27:11.114 INFO:teuthology.orchestra.run.smithi078.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-04-04T09:27:11.160 INFO:teuthology.orchestra.run.smithi078.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-04-04T09:27:11.161 INFO:teuthology.orchestra.run.smithi078.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-04-04T09:27:11.196 INFO:teuthology.orchestra.run.smithi078.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-04-04T09:27:11.197 INFO:teuthology.orchestra.run.smithi078.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-04-04T09:27:11.197 INFO:teuthology.orchestra.run.smithi078.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.3 [28.8 kB] 2021-04-04T09:27:11.197 INFO:teuthology.orchestra.run.smithi078.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-04-04T09:27:11.199 INFO:teuthology.orchestra.run.smithi122.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-04-04T09:27:11.200 INFO:teuthology.orchestra.run.smithi078.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-04-04T09:27:11.200 INFO:teuthology.orchestra.run.smithi078.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-04-04T09:27:11.201 INFO:teuthology.orchestra.run.smithi078.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-04-04T09:27:11.215 INFO:teuthology.orchestra.run.smithi122.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-04-04T09:27:11.231 INFO:teuthology.orchestra.run.smithi122.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-04-04T09:27:11.274 INFO:teuthology.orchestra.run.smithi078.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-04-04T09:27:11.276 INFO:teuthology.orchestra.run.smithi078.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-04-04T09:27:11.287 INFO:teuthology.orchestra.run.smithi122.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-04-04T09:27:11.287 INFO:teuthology.orchestra.run.smithi122.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-04-04T09:27:11.288 INFO:teuthology.orchestra.run.smithi122.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.3 [28.8 kB] 2021-04-04T09:27:11.288 INFO:teuthology.orchestra.run.smithi122.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-04-04T09:27:11.289 INFO:teuthology.orchestra.run.smithi122.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-04-04T09:27:11.289 INFO:teuthology.orchestra.run.smithi122.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-04-04T09:27:11.290 INFO:teuthology.orchestra.run.smithi122.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-04-04T09:27:11.354 INFO:teuthology.orchestra.run.smithi078.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-04-04T09:27:11.355 INFO:teuthology.orchestra.run.smithi078.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-04-04T09:27:11.359 INFO:teuthology.orchestra.run.smithi122.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-04-04T09:27:11.360 INFO:teuthology.orchestra.run.smithi122.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-04-04T09:27:11.361 INFO:teuthology.orchestra.run.smithi078.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-04-04T09:27:11.362 INFO:teuthology.orchestra.run.smithi078.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-04-04T09:27:11.362 INFO:teuthology.orchestra.run.smithi078.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-04-04T09:27:11.365 INFO:teuthology.orchestra.run.smithi078.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-04-04T09:27:11.367 INFO:teuthology.orchestra.run.smithi078.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-04-04T09:27:11.368 INFO:teuthology.orchestra.run.smithi078.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-04-04T09:27:11.434 INFO:teuthology.orchestra.run.smithi078.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-04-04T09:27:11.439 INFO:teuthology.orchestra.run.smithi122.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-04-04T09:27:11.440 INFO:teuthology.orchestra.run.smithi078.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-04-04T09:27:11.440 INFO:teuthology.orchestra.run.smithi122.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-04-04T09:27:11.446 INFO:teuthology.orchestra.run.smithi122.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-04-04T09:27:11.447 INFO:teuthology.orchestra.run.smithi122.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-04-04T09:27:11.447 INFO:teuthology.orchestra.run.smithi122.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-04-04T09:27:11.451 INFO:teuthology.orchestra.run.smithi122.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-04-04T09:27:11.453 INFO:teuthology.orchestra.run.smithi122.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-04-04T09:27:11.454 INFO:teuthology.orchestra.run.smithi122.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-04-04T09:27:11.514 INFO:teuthology.orchestra.run.smithi078.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-04-04T09:27:11.519 INFO:teuthology.orchestra.run.smithi122.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-04-04T09:27:11.521 INFO:teuthology.orchestra.run.smithi078.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-04-04T09:27:11.521 INFO:teuthology.orchestra.run.smithi078.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-04-04T09:27:11.522 INFO:teuthology.orchestra.run.smithi078.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-04-04T09:27:11.523 INFO:teuthology.orchestra.run.smithi078.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-04-04T09:27:11.525 INFO:teuthology.orchestra.run.smithi122.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-04-04T09:27:11.526 INFO:teuthology.orchestra.run.smithi078.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-04-04T09:27:11.528 INFO:teuthology.orchestra.run.smithi078.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-04-04T09:27:11.529 INFO:teuthology.orchestra.run.smithi078.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-04-04T09:27:11.594 INFO:teuthology.orchestra.run.smithi078.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-04-04T09:27:11.595 INFO:teuthology.orchestra.run.smithi078.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-04-04T09:27:11.599 INFO:teuthology.orchestra.run.smithi122.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-04-04T09:27:11.605 INFO:teuthology.orchestra.run.smithi122.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-04-04T09:27:11.606 INFO:teuthology.orchestra.run.smithi122.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-04-04T09:27:11.607 INFO:teuthology.orchestra.run.smithi122.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-04-04T09:27:11.608 INFO:teuthology.orchestra.run.smithi122.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-04-04T09:27:11.610 INFO:teuthology.orchestra.run.smithi122.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-04-04T09:27:11.613 INFO:teuthology.orchestra.run.smithi122.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-04-04T09:27:11.613 INFO:teuthology.orchestra.run.smithi122.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-04-04T09:27:11.674 INFO:teuthology.orchestra.run.smithi078.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.4 [1,097 kB] 2021-04-04T09:27:11.679 INFO:teuthology.orchestra.run.smithi122.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-04-04T09:27:11.680 INFO:teuthology.orchestra.run.smithi122.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-04-04T09:27:11.697 INFO:teuthology.orchestra.run.smithi078.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-04-04T09:27:11.698 INFO:teuthology.orchestra.run.smithi078.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-04-04T09:27:11.698 INFO:teuthology.orchestra.run.smithi078.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-04-04T09:27:11.759 INFO:teuthology.orchestra.run.smithi122.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.4 [1,097 kB] 2021-04-04T09:27:11.774 INFO:teuthology.orchestra.run.smithi122.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-04-04T09:27:11.775 INFO:teuthology.orchestra.run.smithi122.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-04-04T09:27:11.775 INFO:teuthology.orchestra.run.smithi122.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-04-04T09:27:11.837 INFO:teuthology.orchestra.run.smithi078.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-04-04T09:27:11.838 INFO:teuthology.orchestra.run.smithi078.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-04-04T09:27:11.838 INFO:teuthology.orchestra.run.smithi078.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.5 [330 kB] 2021-04-04T09:27:11.846 INFO:teuthology.orchestra.run.smithi078.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-04-04T09:27:11.846 INFO:teuthology.orchestra.run.smithi078.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-04-04T09:27:11.847 INFO:teuthology.orchestra.run.smithi078.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-04-04T09:27:11.848 INFO:teuthology.orchestra.run.smithi078.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-04-04T09:27:11.849 INFO:teuthology.orchestra.run.smithi078.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-04-04T09:27:11.850 INFO:teuthology.orchestra.run.smithi078.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-04-04T09:27:11.867 INFO:teuthology.orchestra.run.smithi122.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-04-04T09:27:11.867 INFO:teuthology.orchestra.run.smithi122.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-04-04T09:27:11.868 INFO:teuthology.orchestra.run.smithi122.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.5 [330 kB] 2021-04-04T09:27:11.873 INFO:teuthology.orchestra.run.smithi122.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-04-04T09:27:11.873 INFO:teuthology.orchestra.run.smithi122.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-04-04T09:27:11.874 INFO:teuthology.orchestra.run.smithi122.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-04-04T09:27:11.919 INFO:teuthology.orchestra.run.smithi122.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-04-04T09:27:11.924 INFO:teuthology.orchestra.run.smithi122.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-04-04T09:27:11.925 INFO:teuthology.orchestra.run.smithi122.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-04-04T09:27:12.014 INFO:teuthology.orchestra.run.smithi078.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.2.0-22-gf4fb4141-1bionic [9,859 kB] 2021-04-04T09:27:12.468 INFO:teuthology.orchestra.run.smithi122.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 16.2.0-22-gf4fb4141-1bionic [9,859 kB] 2021-04-04T09:27:12.608 INFO:teuthology.orchestra.run.smithi078.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.2.0-22-gf4fb4141-1bionic [183 kB] 2021-04-04T09:27:12.612 INFO:teuthology.orchestra.run.smithi078.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 libsqlite3-mod-ceph amd64 16.2.0-22-gf4fb4141-1bionic [134 kB] 2021-04-04T09:27:12.629 INFO:teuthology.orchestra.run.smithi078.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.2.0-22-gf4fb4141-1bionic [1,402 kB] 2021-04-04T09:27:12.707 INFO:teuthology.orchestra.run.smithi078.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.2.0-22-gf4fb4141-1bionic [6,700 kB] 2021-04-04T09:27:13.111 INFO:teuthology.orchestra.run.smithi078.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.2.0-22-gf4fb4141-1bionic [25.3 MB] 2021-04-04T09:27:13.188 INFO:teuthology.orchestra.run.smithi122.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 16.2.0-22-gf4fb4141-1bionic [183 kB] 2021-04-04T09:27:13.193 INFO:teuthology.orchestra.run.smithi122.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 libsqlite3-mod-ceph amd64 16.2.0-22-gf4fb4141-1bionic [134 kB] 2021-04-04T09:27:13.198 INFO:teuthology.orchestra.run.smithi122.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 16.2.0-22-gf4fb4141-1bionic [1,402 kB] 2021-04-04T09:27:13.304 INFO:teuthology.orchestra.run.smithi122.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 16.2.0-22-gf4fb4141-1bionic [6,700 kB] 2021-04-04T09:27:13.796 INFO:teuthology.orchestra.run.smithi122.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 16.2.0-22-gf4fb4141-1bionic [25.3 MB] 2021-04-04T09:27:14.636 INFO:teuthology.orchestra.run.smithi078.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.2.0-22-gf4fb4141-1bionic [3,840 B] 2021-04-04T09:27:14.636 INFO:teuthology.orchestra.run.smithi078.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.2.0-22-gf4fb4141-1bionic [782 kB] 2021-04-04T09:27:14.676 INFO:teuthology.orchestra.run.smithi078.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.2.0-22-gf4fb4141-1bionic [2,167 kB] 2021-04-04T09:27:14.816 INFO:teuthology.orchestra.run.smithi078.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.2.0-22-gf4fb4141-1bionic [63.6 kB] 2021-04-04T09:27:14.818 INFO:teuthology.orchestra.run.smithi078.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.2.0-22-gf4fb4141-1bionic [75.6 kB] 2021-04-04T09:27:14.821 INFO:teuthology.orchestra.run.smithi078.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.2.0-22-gf4fb4141-1bionic [2,335 kB] 2021-04-04T09:27:14.965 INFO:teuthology.orchestra.run.smithi078.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.2.0-22-gf4fb4141-1bionic [7,874 kB] 2021-04-04T09:27:15.439 INFO:teuthology.orchestra.run.smithi078.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.2.0-22-gf4fb4141-1bionic [3,956 B] 2021-04-04T09:27:15.440 INFO:teuthology.orchestra.run.smithi078.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.2.0-22-gf4fb4141-1bionic [22.1 kB] 2021-04-04T09:27:15.440 INFO:teuthology.orchestra.run.smithi078.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.2.0-22-gf4fb4141-1bionic [46.7 MB] 2021-04-04T09:27:15.626 INFO:teuthology.orchestra.run.smithi122.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 16.2.0-22-gf4fb4141-1bionic [3,840 B] 2021-04-04T09:27:15.627 INFO:teuthology.orchestra.run.smithi122.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 16.2.0-22-gf4fb4141-1bionic [782 kB] 2021-04-04T09:27:15.665 INFO:teuthology.orchestra.run.smithi122.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 16.2.0-22-gf4fb4141-1bionic [2,167 kB] 2021-04-04T09:27:15.814 INFO:teuthology.orchestra.run.smithi122.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 16.2.0-22-gf4fb4141-1bionic [63.6 kB] 2021-04-04T09:27:15.816 INFO:teuthology.orchestra.run.smithi122.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 16.2.0-22-gf4fb4141-1bionic [75.6 kB] 2021-04-04T09:27:15.821 INFO:teuthology.orchestra.run.smithi122.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 16.2.0-22-gf4fb4141-1bionic [2,335 kB] 2021-04-04T09:27:15.973 INFO:teuthology.orchestra.run.smithi122.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 16.2.0-22-gf4fb4141-1bionic [7,874 kB] 2021-04-04T09:27:16.452 INFO:teuthology.orchestra.run.smithi122.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 16.2.0-22-gf4fb4141-1bionic [3,956 B] 2021-04-04T09:27:16.453 INFO:teuthology.orchestra.run.smithi122.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 16.2.0-22-gf4fb4141-1bionic [22.1 kB] 2021-04-04T09:27:16.454 INFO:teuthology.orchestra.run.smithi122.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 16.2.0-22-gf4fb4141-1bionic [46.7 MB] 2021-04-04T09:27:18.650 INFO:teuthology.orchestra.run.smithi078.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.2.0-22-gf4fb4141-1bionic [20.2 kB] 2021-04-04T09:27:18.650 INFO:teuthology.orchestra.run.smithi078.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.2.0-22-gf4fb4141-1bionic [10.6 MB] 2021-04-04T09:27:19.364 INFO:teuthology.orchestra.run.smithi078.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.2.0-22-gf4fb4141-1bionic [87.2 kB] 2021-04-04T09:27:19.368 INFO:teuthology.orchestra.run.smithi078.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 rbd-mirror amd64 16.2.0-22-gf4fb4141-1bionic [3,344 kB] 2021-04-04T09:27:19.380 INFO:teuthology.orchestra.run.smithi122.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 16.2.0-22-gf4fb4141-1bionic [20.2 kB] 2021-04-04T09:27:19.381 INFO:teuthology.orchestra.run.smithi122.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 16.2.0-22-gf4fb4141-1bionic [10.6 MB] 2021-04-04T09:27:19.831 INFO:teuthology.orchestra.run.smithi122.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 16.2.0-22-gf4fb4141-1bionic [87.2 kB] 2021-04-04T09:27:19.833 INFO:teuthology.orchestra.run.smithi122.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/pacific/f4fb41419a42bee43a2069637ef17064073bf30b/ubuntu/bionic/flavors/default bionic/main amd64 rbd-mirror amd64 16.2.0-22-gf4fb4141-1bionic [3,344 kB] 2021-04-04T09:27:20.299 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 178 MB in 10s (18.0 MB/s) 2021-04-04T09:27:20.448 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-uritemplate. 2021-04-04T09:27:20.503 INFO:teuthology.orchestra.run.smithi078.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137596 files and directories currently installed.) 2021-04-04T09:27:20.504 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-04-04T09:27:20.538 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-04-04T09:27:20.648 INFO:teuthology.orchestra.run.smithi122.stdout:Fetched 178 MB in 10s (17.5 MB/s) 2021-04-04T09:27:20.783 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libjaeger. 2021-04-04T09:27:20.791 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-uritemplate. 2021-04-04T09:27:20.796 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../001-libjaeger_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:20.832 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libjaeger (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:20.839 INFO:teuthology.orchestra.run.smithi122.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137596 files and directories currently installed.) 2021-04-04T09:27:20.843 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-04-04T09:27:20.889 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-04-04T09:27:21.133 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libjaeger. 2021-04-04T09:27:21.148 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../001-libjaeger_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:21.192 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libjaeger (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:21.269 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-04-04T09:27:21.283 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../002-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-04-04T09:27:21.369 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-04-04T09:27:21.486 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-04-04T09:27:21.502 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../002-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-04-04T09:27:21.586 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-04-04T09:27:21.711 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../003-librbd1_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:21.832 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librbd1 (16.2.0-22-gf4fb4141-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-04-04T09:27:21.912 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../003-librbd1_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:22.040 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librbd1 (16.2.0-22-gf4fb4141-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-04-04T09:27:22.540 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../004-librados2_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:22.668 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librados2 (16.2.0-22-gf4fb4141-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-04-04T09:27:22.700 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../004-librados2_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:22.828 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librados2 (16.2.0-22-gf4fb4141-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-04-04T09:27:23.331 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libcephfs2. 2021-04-04T09:27:23.346 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../005-libcephfs2_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:23.380 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libcephfs2 (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:23.465 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libcephfs2. 2021-04-04T09:27:23.480 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../005-libcephfs2_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:23.523 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libcephfs2 (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:23.683 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rados. 2021-04-04T09:27:23.698 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../006-python3-rados_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:23.741 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rados (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:23.842 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-rados. 2021-04-04T09:27:23.855 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../006-python3-rados_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:23.892 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-rados (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:24.111 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-04-04T09:27:24.124 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../007-python3-ceph-argparse_16.2.0-22-gf4fb4141-1bionic_all.deb ... 2021-04-04T09:27:24.169 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-ceph-argparse (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:24.245 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-04-04T09:27:24.258 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../007-python3-ceph-argparse_16.2.0-22-gf4fb4141-1bionic_all.deb ... 2021-04-04T09:27:24.295 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-ceph-argparse (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:24.488 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cephfs. 2021-04-04T09:27:24.504 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../008-python3-cephfs_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:24.555 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cephfs (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:24.598 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cephfs. 2021-04-04T09:27:24.612 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../008-python3-cephfs_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:24.656 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cephfs (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:24.866 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-ceph-common. 2021-04-04T09:27:24.885 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../009-python3-ceph-common_16.2.0-22-gf4fb4141-1bionic_all.deb ... 2021-04-04T09:27:24.924 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-ceph-common (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:24.992 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-ceph-common. 2021-04-04T09:27:25.005 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../009-python3-ceph-common_16.2.0-22-gf4fb4141-1bionic_all.deb ... 2021-04-04T09:27:25.042 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-ceph-common (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:25.252 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-prettytable. 2021-04-04T09:27:25.266 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../010-python3-prettytable_0.7.2-3_all.deb ... 2021-04-04T09:27:25.310 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-04-04T09:27:25.362 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-prettytable. 2021-04-04T09:27:25.377 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../010-python3-prettytable_0.7.2-3_all.deb ... 2021-04-04T09:27:25.420 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-04-04T09:27:25.571 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rbd. 2021-04-04T09:27:25.586 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../011-python3-rbd_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:25.629 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rbd (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:25.664 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-rbd. 2021-04-04T09:27:25.677 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../011-python3-rbd_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:25.714 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-rbd (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:25.991 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-04-04T09:27:26.004 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../012-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-04-04T09:27:26.042 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-04-04T09:27:26.300 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-04-04T09:27:26.314 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../012-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-04-04T09:27:26.350 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-04-04T09:27:26.543 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-04-04T09:27:26.556 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../013-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-04-04T09:27:26.593 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-04-04T09:27:26.678 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-04-04T09:27:26.692 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../013-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-04-04T09:27:26.736 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-04-04T09:27:26.929 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librgw2. 2021-04-04T09:27:26.943 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../014-librgw2_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:26.996 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librgw2 (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:27.064 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package librgw2. 2021-04-04T09:27:27.077 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../014-librgw2_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:27.114 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking librgw2 (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:27.565 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rgw. 2021-04-04T09:27:27.579 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../015-python3-rgw_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:27.623 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rgw (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:27.700 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-rgw. 2021-04-04T09:27:27.716 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../015-python3-rgw_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:27.758 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-rgw (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:27.925 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-04-04T09:27:27.941 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-04-04T09:27:27.984 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-04-04T09:27:28.078 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-04-04T09:27:28.092 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../016-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-04-04T09:27:28.136 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-04-04T09:27:28.270 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-04-04T09:27:28.285 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2021-04-04T09:27:28.328 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-04-04T09:27:28.430 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-04-04T09:27:28.445 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../017-libleveldb1v5_1.20-2_amd64.deb ... 2021-04-04T09:27:28.489 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-04-04T09:27:28.723 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package liboath0. 2021-04-04T09:27:28.736 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2021-04-04T09:27:28.773 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-04-04T09:27:28.833 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package liboath0. 2021-04-04T09:27:28.848 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../018-liboath0_2.6.1-1_amd64.deb ... 2021-04-04T09:27:28.889 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-04-04T09:27:29.092 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libradosstriper1. 2021-04-04T09:27:29.105 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../019-libradosstriper1_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:29.142 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libradosstriper1 (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:29.178 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libradosstriper1. 2021-04-04T09:27:29.196 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../019-libradosstriper1_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:29.236 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libradosstriper1 (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:29.587 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-common. 2021-04-04T09:27:29.600 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../020-ceph-common_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:29.637 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-common (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:29.689 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-common. 2021-04-04T09:27:29.703 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../020-ceph-common_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:29.747 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-common (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:31.650 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-base. 2021-04-04T09:27:31.664 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../021-ceph-base_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:31.718 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-base (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:31.719 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-base. 2021-04-04T09:27:31.732 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../021-ceph-base_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:31.779 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-base (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:32.930 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-dateutil. 2021-04-04T09:27:32.944 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../022-python3-dateutil_2.6.1-1_all.deb ... 2021-04-04T09:27:33.005 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-04-04T09:27:33.049 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-dateutil. 2021-04-04T09:27:33.063 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../022-python3-dateutil_2.6.1-1_all.deb ... 2021-04-04T09:27:33.107 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-04-04T09:27:33.341 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-asn1crypto. 2021-04-04T09:27:33.356 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../023-python3-asn1crypto_0.24.0-1_all.deb ... 2021-04-04T09:27:33.399 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-04-04T09:27:33.419 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-asn1crypto. 2021-04-04T09:27:33.434 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../023-python3-asn1crypto_0.24.0-1_all.deb ... 2021-04-04T09:27:33.477 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-04-04T09:27:33.677 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cffi-backend. 2021-04-04T09:27:33.691 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../024-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-04-04T09:27:33.735 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-04-04T09:27:33.746 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cffi-backend. 2021-04-04T09:27:33.760 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../024-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-04-04T09:27:33.804 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-04-04T09:27:34.021 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cryptography. 2021-04-04T09:27:34.037 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../025-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-04-04T09:27:34.079 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-04-04T09:27:34.123 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cryptography. 2021-04-04T09:27:34.139 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../025-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-04-04T09:27:34.182 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-04-04T09:27:34.491 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-openssl. 2021-04-04T09:27:34.508 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../026-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-04-04T09:27:34.541 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-04-04T09:27:34.577 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-openssl. 2021-04-04T09:27:34.591 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../026-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-04-04T09:27:34.635 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-04-04T09:27:34.802 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-04-04T09:27:34.814 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_16.2.0-22-gf4fb4141-1bionic_all.deb ... 2021-04-04T09:27:34.852 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-modules-core (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:34.871 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-04-04T09:27:34.884 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../027-ceph-mgr-modules-core_16.2.0-22-gf4fb4141-1bionic_all.deb ... 2021-04-04T09:27:34.912 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-modules-core (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:35.164 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2021-04-04T09:27:35.179 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../028-libsqlite3-mod-ceph_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:35.192 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2021-04-04T09:27:35.200 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../028-libsqlite3-mod-ceph_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:35.223 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libsqlite3-mod-ceph (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:35.234 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libsqlite3-mod-ceph (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:35.550 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-bcrypt. 2021-04-04T09:27:35.561 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-bcrypt. 2021-04-04T09:27:35.564 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-04-04T09:27:35.574 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-04-04T09:27:35.611 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-04-04T09:27:35.617 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-04-04T09:27:35.936 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cherrypy3. 2021-04-04T09:27:35.951 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-2_all.deb ... 2021-04-04T09:27:35.955 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cherrypy3. 2021-04-04T09:27:35.969 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-2_all.deb ... 2021-04-04T09:27:36.011 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-04-04T09:27:36.014 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-04-04T09:27:36.333 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jwt. 2021-04-04T09:27:36.349 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../031-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-04-04T09:27:36.373 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jwt. 2021-04-04T09:27:36.387 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../031-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-04-04T09:27:36.392 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-04-04T09:27:36.431 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-04-04T09:27:36.742 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-logutils. 2021-04-04T09:27:36.756 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-5_all.deb ... 2021-04-04T09:27:36.762 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-logutils. 2021-04-04T09:27:36.776 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../032-python3-logutils_0.3.3-5_all.deb ... 2021-04-04T09:27:36.800 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-04-04T09:27:36.820 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-04-04T09:27:37.103 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-markupsafe. 2021-04-04T09:27:37.106 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-markupsafe. 2021-04-04T09:27:37.117 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../033-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-04-04T09:27:37.121 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../033-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-04-04T09:27:37.153 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-04-04T09:27:37.164 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-04-04T09:27:37.456 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-mako. 2021-04-04T09:27:37.467 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-mako. 2021-04-04T09:27:37.470 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../034-python3-mako_1.0.7+ds1-1_all.deb ... 2021-04-04T09:27:37.480 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../034-python3-mako_1.0.7+ds1-1_all.deb ... 2021-04-04T09:27:37.517 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-04-04T09:27:37.522 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-04-04T09:27:37.845 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-simplegeneric. 2021-04-04T09:27:37.860 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-1_all.deb ... 2021-04-04T09:27:37.875 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-simplegeneric. 2021-04-04T09:27:37.888 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../035-python3-simplegeneric_0.8.1-1_all.deb ... 2021-04-04T09:27:37.903 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-04-04T09:27:37.927 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-04-04T09:27:38.206 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-singledispatch. 2021-04-04T09:27:38.220 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-04-04T09:27:38.228 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-singledispatch. 2021-04-04T09:27:38.244 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../036-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-04-04T09:27:38.273 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-04-04T09:27:38.286 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-04-04T09:27:38.583 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-webob. 2021-04-04T09:27:38.588 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-webob. 2021-04-04T09:27:38.597 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../037-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-04-04T09:27:38.603 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../037-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-04-04T09:27:38.633 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-04-04T09:27:38.655 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-04-04T09:27:38.953 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-bs4. 2021-04-04T09:27:38.967 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../038-python3-bs4_4.6.0-1_all.deb ... 2021-04-04T09:27:39.008 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-bs4. 2021-04-04T09:27:39.011 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-04-04T09:27:39.024 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../038-python3-bs4_4.6.0-1_all.deb ... 2021-04-04T09:27:39.066 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-04-04T09:27:39.322 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-waitress. 2021-04-04T09:27:39.337 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../039-python3-waitress_1.0.1-1_all.deb ... 2021-04-04T09:27:39.380 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-04-04T09:27:39.386 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-waitress. 2021-04-04T09:27:39.399 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../039-python3-waitress_1.0.1-1_all.deb ... 2021-04-04T09:27:39.444 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-04-04T09:27:39.700 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-tempita. 2021-04-04T09:27:39.719 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../040-python3-tempita_0.5.2-2_all.deb ... 2021-04-04T09:27:39.750 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-04-04T09:27:39.763 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-tempita. 2021-04-04T09:27:39.777 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../040-python3-tempita_0.5.2-2_all.deb ... 2021-04-04T09:27:39.830 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-04-04T09:27:40.061 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-paste. 2021-04-04T09:27:40.075 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../041-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-04-04T09:27:40.118 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-04-04T09:27:40.159 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-paste. 2021-04-04T09:27:40.174 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../041-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-04-04T09:27:40.216 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-04-04T09:27:40.456 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-04-04T09:27:40.473 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../042-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-04-04T09:27:40.514 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-04-04T09:27:40.552 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-04-04T09:27:40.567 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../042-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-04-04T09:27:40.611 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-04-04T09:27:40.817 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pastedeploy. 2021-04-04T09:27:40.832 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../043-python3-pastedeploy_1.5.2-4_all.deb ... 2021-04-04T09:27:40.875 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-04-04T09:27:40.913 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pastedeploy. 2021-04-04T09:27:40.927 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../043-python3-pastedeploy_1.5.2-4_all.deb ... 2021-04-04T09:27:40.971 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-04-04T09:27:41.178 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-webtest. 2021-04-04T09:27:41.192 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../044-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-04-04T09:27:41.236 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-04-04T09:27:41.283 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-webtest. 2021-04-04T09:27:41.296 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../044-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-04-04T09:27:41.333 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-04-04T09:27:42.406 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pecan. 2021-04-04T09:27:42.411 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pecan. 2021-04-04T09:27:42.420 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../045-python3-pecan_1.2.1-2_all.deb ... 2021-04-04T09:27:42.424 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../045-python3-pecan_1.2.1-2_all.deb ... 2021-04-04T09:27:42.464 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-04-04T09:27:42.465 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-04-04T09:27:42.832 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-werkzeug. 2021-04-04T09:27:42.845 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../046-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-04-04T09:27:42.869 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-werkzeug. 2021-04-04T09:27:42.882 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-04-04T09:27:42.883 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../046-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-04-04T09:27:42.927 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-04-04T09:27:43.243 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr. 2021-04-04T09:27:43.256 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../047-ceph-mgr_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:43.307 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:43.331 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr. 2021-04-04T09:27:43.347 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../047-ceph-mgr_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:43.390 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:43.794 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mon. 2021-04-04T09:27:43.810 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../048-ceph-mon_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:43.852 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mon (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:43.876 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mon. 2021-04-04T09:27:43.889 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../048-ceph-mon_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:43.925 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mon (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:44.789 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-osd. 2021-04-04T09:27:44.803 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../049-ceph-osd_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:44.847 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-osd (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:44.879 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-osd. 2021-04-04T09:27:44.892 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../049-ceph-osd_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:44.937 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-osd (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:46.894 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph. 2021-04-04T09:27:46.906 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../050-ceph_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:46.944 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:46.992 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph. 2021-04-04T09:27:47.005 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../050-ceph_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:47.042 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:47.262 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-fuse. 2021-04-04T09:27:47.276 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../051-ceph-fuse_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:47.312 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-fuse (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:47.361 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-fuse. 2021-04-04T09:27:47.376 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../051-ceph-fuse_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:47.419 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-fuse (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:47.741 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mds. 2021-04-04T09:27:47.755 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../052-ceph-mds_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:47.799 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mds (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:47.888 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mds. 2021-04-04T09:27:47.903 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../052-ceph-mds_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:47.946 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mds (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:48.358 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package cephadm. 2021-04-04T09:27:48.372 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../053-cephadm_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:48.416 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking cephadm (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:48.484 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package cephadm. 2021-04-04T09:27:48.496 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../053-cephadm_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:27:48.533 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking cephadm (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:48.736 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jinja2. 2021-04-04T09:27:48.752 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../054-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-04-04T09:27:48.794 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-04-04T09:27:48.853 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jinja2. 2021-04-04T09:27:48.869 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../054-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-04-04T09:27:48.912 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-04-04T09:27:49.113 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-04-04T09:27:49.129 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../055-ceph-mgr-cephadm_16.2.0-22-gf4fb4141-1bionic_all.deb ... 2021-04-04T09:27:49.172 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-cephadm (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:49.214 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-04-04T09:27:49.230 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../055-ceph-mgr-cephadm_16.2.0-22-gf4fb4141-1bionic_all.deb ... 2021-04-04T09:27:49.273 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-cephadm (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:49.492 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-repoze.lru. 2021-04-04T09:27:49.508 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../056-python3-repoze.lru_0.7-1_all.deb ... 2021-04-04T09:27:49.550 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-04-04T09:27:49.618 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-repoze.lru. 2021-04-04T09:27:49.632 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../056-python3-repoze.lru_0.7-1_all.deb ... 2021-04-04T09:27:49.676 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-04-04T09:27:49.903 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-routes. 2021-04-04T09:27:49.916 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../057-python3-routes_2.4.1-1_all.deb ... 2021-04-04T09:27:49.953 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-04-04T09:27:49.996 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-routes. 2021-04-04T09:27:50.012 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../057-python3-routes_2.4.1-1_all.deb ... 2021-04-04T09:27:50.062 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-04-04T09:27:50.264 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-04-04T09:27:50.279 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../058-ceph-mgr-dashboard_16.2.0-22-gf4fb4141-1bionic_all.deb ... 2021-04-04T09:27:50.314 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-dashboard (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:50.390 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-04-04T09:27:50.404 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../058-ceph-mgr-dashboard_16.2.0-22-gf4fb4141-1bionic_all.deb ... 2021-04-04T09:27:50.450 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-dashboard (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:51.211 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-numpy. 2021-04-04T09:27:51.229 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../059-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-04-04T09:27:51.271 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-04-04T09:27:51.342 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-numpy. 2021-04-04T09:27:51.357 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../059-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-04-04T09:27:51.401 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-04-04T09:27:51.901 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-decorator. 2021-04-04T09:27:51.916 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../060-python3-decorator_4.1.2-1_all.deb ... 2021-04-04T09:27:51.959 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-04-04T09:27:52.028 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-decorator. 2021-04-04T09:27:52.043 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../060-python3-decorator_4.1.2-1_all.deb ... 2021-04-04T09:27:52.087 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-04-04T09:27:52.254 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-scipy. 2021-04-04T09:27:52.268 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../061-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-04-04T09:27:52.312 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-04-04T09:27:52.406 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-scipy. 2021-04-04T09:27:52.420 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../061-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-04-04T09:27:52.473 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-04-04T09:27:53.630 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-04-04T09:27:53.645 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-04-04T09:27:53.688 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-04-04T09:27:53.792 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-04-04T09:27:53.808 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-04-04T09:27:53.850 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-04-04T09:27:54.184 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-joblib. 2021-04-04T09:27:54.199 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../063-python3-joblib_0.11-1_all.deb ... 2021-04-04T09:27:54.242 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-joblib (0.11-1) ... 2021-04-04T09:27:54.304 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-joblib. 2021-04-04T09:27:54.319 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../063-python3-joblib_0.11-1_all.deb ... 2021-04-04T09:27:54.363 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-joblib (0.11-1) ... 2021-04-04T09:27:54.595 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-sklearn. 2021-04-04T09:27:54.608 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../064-python3-sklearn_0.19.1-3_all.deb ... 2021-04-04T09:27:54.645 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-04-04T09:27:54.726 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-sklearn. 2021-04-04T09:27:54.742 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../064-python3-sklearn_0.19.1-3_all.deb ... 2021-04-04T09:27:54.784 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-04-04T09:27:55.175 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-04-04T09:27:55.188 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_16.2.0-22-gf4fb4141-1bionic_all.deb ... 2021-04-04T09:27:55.233 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-diskprediction-local (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:55.366 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-04-04T09:27:55.381 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_16.2.0-22-gf4fb4141-1bionic_all.deb ... 2021-04-04T09:27:55.425 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-diskprediction-local (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:27:56.328 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-monotonic. 2021-04-04T09:27:56.343 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../066-python3-monotonic_1.1-2_all.deb ... 2021-04-04T09:27:56.386 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-04-04T09:27:56.545 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-monotonic. 2021-04-04T09:27:56.560 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../066-python3-monotonic_1.1-2_all.deb ... 2021-04-04T09:27:56.603 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-04-04T09:27:56.698 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-fasteners. 2021-04-04T09:27:56.712 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../067-python3-fasteners_0.12.0-3_all.deb ... 2021-04-04T09:27:56.756 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-04-04T09:27:57.265 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-fasteners. 2021-04-04T09:27:57.280 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../067-python3-fasteners_0.12.0-3_all.deb ... 2021-04-04T09:27:57.323 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-04-04T09:27:57.484 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-httplib2. 2021-04-04T09:27:57.499 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../068-python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb ... 2021-04-04T09:27:57.542 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-04-04T09:27:57.651 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-httplib2. 2021-04-04T09:27:57.665 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../068-python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb ... 2021-04-04T09:27:57.709 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-04-04T09:27:57.828 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyasn1. 2021-04-04T09:27:57.845 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../069-python3-pyasn1_0.4.2-3_all.deb ... 2021-04-04T09:27:57.887 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-04-04T09:27:58.013 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pyasn1. 2021-04-04T09:27:58.026 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../069-python3-pyasn1_0.4.2-3_all.deb ... 2021-04-04T09:27:58.071 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-04-04T09:27:58.240 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-04-04T09:27:58.255 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../070-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-04-04T09:27:58.305 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-04-04T09:27:58.417 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-04-04T09:27:58.432 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../070-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-04-04T09:27:58.483 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-04-04T09:27:58.610 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rsa. 2021-04-04T09:27:58.624 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../071-python3-rsa_3.4.2-1_all.deb ... 2021-04-04T09:27:58.668 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-04-04T09:27:58.836 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-rsa. 2021-04-04T09:27:58.852 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../071-python3-rsa_3.4.2-1_all.deb ... 2021-04-04T09:27:58.903 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-04-04T09:27:59.005 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-04-04T09:27:59.022 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../072-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-04-04T09:27:59.063 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-04-04T09:27:59.215 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-04-04T09:27:59.229 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../072-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-04-04T09:27:59.273 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-04-04T09:27:59.493 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-oauth2client. 2021-04-04T09:27:59.509 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../073-python3-oauth2client_4.1.2-3_all.deb ... 2021-04-04T09:27:59.551 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-04-04T09:27:59.737 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-oauth2client. 2021-04-04T09:27:59.753 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../073-python3-oauth2client_4.1.2-3_all.deb ... 2021-04-04T09:27:59.820 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-04-04T09:27:59.887 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-setuptools. 2021-04-04T09:27:59.906 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../074-python3-setuptools_39.0.1-2_all.deb ... 2021-04-04T09:27:59.945 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-04-04T09:28:00.198 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-setuptools. 2021-04-04T09:28:00.213 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../074-python3-setuptools_39.0.1-2_all.deb ... 2021-04-04T09:28:00.256 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-04-04T09:28:00.374 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-websocket. 2021-04-04T09:28:00.390 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../075-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-04-04T09:28:00.432 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-04-04T09:28:00.652 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-websocket. 2021-04-04T09:28:00.668 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../075-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-04-04T09:28:00.710 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-04-04T09:28:00.744 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-kubernetes. 2021-04-04T09:28:00.762 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../076-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-04-04T09:28:00.802 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-04-04T09:28:01.072 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-kubernetes. 2021-04-04T09:28:01.087 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../076-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-04-04T09:28:01.147 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-04-04T09:28:01.215 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-04-04T09:28:01.232 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../077-ceph-mgr-k8sevents_16.2.0-22-gf4fb4141-1bionic_all.deb ... 2021-04-04T09:28:01.273 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-k8sevents (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:01.577 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-04-04T09:28:01.590 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../077-ceph-mgr-k8sevents_16.2.0-22-gf4fb4141-1bionic_all.deb ... 2021-04-04T09:28:01.626 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-json-pointer. 2021-04-04T09:28:01.627 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-k8sevents (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:01.640 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../078-python3-json-pointer_1.10-1_all.deb ... 2021-04-04T09:28:01.692 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-04-04T09:28:01.948 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-json-pointer. 2021-04-04T09:28:01.964 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../078-python3-json-pointer_1.10-1_all.deb ... 2021-04-04T09:28:02.005 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-04-04T09:28:02.071 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jsonpatch. 2021-04-04T09:28:02.086 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../079-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-04-04T09:28:02.132 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-04-04T09:28:02.383 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-jsonpatch. 2021-04-04T09:28:02.397 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../079-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-04-04T09:28:02.432 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-04-04T09:28:02.447 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../080-ceph-mgr-rook_16.2.0-22-gf4fb4141-1bionic_all.deb ... 2021-04-04T09:28:02.452 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-04-04T09:28:02.567 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-rook (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:02.753 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-04-04T09:28:02.769 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../080-ceph-mgr-rook_16.2.0-22-gf4fb4141-1bionic_all.deb ... 2021-04-04T09:28:02.888 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-mgr-rook (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:02.912 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libonig4:amd64. 2021-04-04T09:28:02.927 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../081-libonig4_6.7.0-1_amd64.deb ... 2021-04-04T09:28:02.970 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-04-04T09:28:03.216 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libonig4:amd64. 2021-04-04T09:28:03.230 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../081-libonig4_6.7.0-1_amd64.deb ... 2021-04-04T09:28:03.266 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-04-04T09:28:03.340 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libjq1:amd64. 2021-04-04T09:28:03.357 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../082-libjq1_1.5+dfsg-2_amd64.deb ... 2021-04-04T09:28:03.398 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-04-04T09:28:03.645 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libjq1:amd64. 2021-04-04T09:28:03.660 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../082-libjq1_1.5+dfsg-2_amd64.deb ... 2021-04-04T09:28:03.667 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package jq. 2021-04-04T09:28:03.683 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../083-jq_1.5+dfsg-2_amd64.deb ... 2021-04-04T09:28:03.703 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-04-04T09:28:03.726 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-04-04T09:28:03.981 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package jq. 2021-04-04T09:28:03.995 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../083-jq_1.5+dfsg-2_amd64.deb ... 2021-04-04T09:28:03.995 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package socat. 2021-04-04T09:28:04.011 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../084-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-04-04T09:28:04.031 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-04-04T09:28:04.054 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-04-04T09:28:04.284 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package socat. 2021-04-04T09:28:04.297 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../084-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-04-04T09:28:04.334 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-04-04T09:28:04.474 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-04-04T09:28:04.488 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../085-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-04-04T09:28:04.532 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-04-04T09:28:04.755 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-04-04T09:28:04.769 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../085-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-04-04T09:28:04.821 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-04-04T09:28:04.844 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package xmlstarlet. 2021-04-04T09:28:04.857 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2_amd64.deb ... 2021-04-04T09:28:04.902 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-04-04T09:28:05.083 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package xmlstarlet. 2021-04-04T09:28:05.099 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2_amd64.deb ... 2021-04-04T09:28:05.141 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-04-04T09:28:05.273 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-test. 2021-04-04T09:28:05.288 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../087-ceph-test_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:28:05.331 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-test (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:05.504 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package ceph-test. 2021-04-04T09:28:05.518 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../087-ceph-test_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:28:05.554 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking ceph-test (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:08.760 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libcephfs-dev. 2021-04-04T09:28:08.776 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../088-libcephfs-dev_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:28:08.818 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libcephfs-dev (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:09.066 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package libcephfs-dev. 2021-04-04T09:28:09.083 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../088-libcephfs-dev_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:28:09.113 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package nvme-cli. 2021-04-04T09:28:09.125 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking libcephfs-dev (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:09.127 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../089-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-04-04T09:28:09.171 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-04-04T09:28:09.420 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package nvme-cli. 2021-04-04T09:28:09.436 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../089-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-04-04T09:28:09.478 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-04-04T09:28:09.542 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-attr. 2021-04-04T09:28:09.563 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../090-python-attr_17.4.0-2_all.deb ... 2021-04-04T09:28:09.600 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-attr (17.4.0-2) ... 2021-04-04T09:28:09.824 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python-attr. 2021-04-04T09:28:09.839 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../090-python-attr_17.4.0-2_all.deb ... 2021-04-04T09:28:09.883 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python-attr (17.4.0-2) ... 2021-04-04T09:28:09.903 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-funcsigs. 2021-04-04T09:28:09.917 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../091-python-funcsigs_1.0.2-4_all.deb ... 2021-04-04T09:28:09.978 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-04-04T09:28:10.211 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python-funcsigs. 2021-04-04T09:28:10.228 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../091-python-funcsigs_1.0.2-4_all.deb ... 2021-04-04T09:28:10.269 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-04-04T09:28:10.298 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-pluggy. 2021-04-04T09:28:10.312 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../092-python-pluggy_0.6.0-1_all.deb ... 2021-04-04T09:28:10.356 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-04-04T09:28:10.581 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python-pluggy. 2021-04-04T09:28:10.598 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../092-python-pluggy_0.6.0-1_all.deb ... 2021-04-04T09:28:10.639 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-04-04T09:28:10.659 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-py. 2021-04-04T09:28:10.674 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../093-python-py_1.5.2-1_all.deb ... 2021-04-04T09:28:10.718 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-py (1.5.2-1) ... 2021-04-04T09:28:10.942 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python-py. 2021-04-04T09:28:10.957 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../093-python-py_1.5.2-1_all.deb ... 2021-04-04T09:28:11.017 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python-py (1.5.2-1) ... 2021-04-04T09:28:11.089 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-pytest. 2021-04-04T09:28:11.102 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../094-python-pytest_3.3.2-2_all.deb ... 2021-04-04T09:28:11.147 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-04-04T09:28:11.338 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python-pytest. 2021-04-04T09:28:11.354 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../094-python-pytest_3.3.2-2_all.deb ... 2021-04-04T09:28:11.396 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-04-04T09:28:11.484 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-attr. 2021-04-04T09:28:11.500 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../095-python3-attr_17.4.0-2_all.deb ... 2021-04-04T09:28:11.542 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-04-04T09:28:11.767 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-attr. 2021-04-04T09:28:11.781 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../095-python3-attr_17.4.0-2_all.deb ... 2021-04-04T09:28:11.825 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-04-04T09:28:11.870 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cycler. 2021-04-04T09:28:11.884 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../096-python3-cycler_0.10.0-1_all.deb ... 2021-04-04T09:28:11.928 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-04-04T09:28:12.137 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-cycler. 2021-04-04T09:28:12.152 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../096-python3-cycler_0.10.0-1_all.deb ... 2021-04-04T09:28:12.195 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-04-04T09:28:12.240 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-webencodings. 2021-04-04T09:28:12.257 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../097-python3-webencodings_0.5-2_all.deb ... 2021-04-04T09:28:12.298 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-04-04T09:28:12.506 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-webencodings. 2021-04-04T09:28:12.521 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../097-python3-webencodings_0.5-2_all.deb ... 2021-04-04T09:28:12.565 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-04-04T09:28:12.621 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-html5lib. 2021-04-04T09:28:12.632 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../098-python3-html5lib_0.999999999-1_all.deb ... 2021-04-04T09:28:13.877 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-04-04T09:28:13.877 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-04-04T09:28:13.878 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../099-python3-lxml_4.2.1-1ubuntu0.4_amd64.deb ... 2021-04-04T09:28:13.878 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.4) ... 2021-04-04T09:28:13.879 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyparsing. 2021-04-04T09:28:13.879 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../100-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-04-04T09:28:13.880 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-04-04T09:28:13.880 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-html5lib. 2021-04-04T09:28:13.881 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../098-python3-html5lib_0.999999999-1_all.deb ... 2021-04-04T09:28:13.881 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-04-04T09:28:13.881 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-04-04T09:28:13.882 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../099-python3-lxml_4.2.1-1ubuntu0.4_amd64.deb ... 2021-04-04T09:28:13.882 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.4) ... 2021-04-04T09:28:13.883 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pyparsing. 2021-04-04T09:28:13.883 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../100-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-04-04T09:28:13.910 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-04-04T09:28:13.920 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-tz. 2021-04-04T09:28:13.932 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../101-python3-tz_2018.3-2_all.deb ... 2021-04-04T09:28:13.977 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-tz (2018.3-2) ... 2021-04-04T09:28:14.245 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-tz. 2021-04-04T09:28:14.260 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../101-python3-tz_2018.3-2_all.deb ... 2021-04-04T09:28:14.280 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-matplotlib. 2021-04-04T09:28:14.295 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../102-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-04-04T09:28:14.303 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-tz (2018.3-2) ... 2021-04-04T09:28:14.338 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-04-04T09:28:14.615 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-matplotlib. 2021-04-04T09:28:14.629 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../102-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-04-04T09:28:14.673 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-04-04T09:28:15.002 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-olefile. 2021-04-04T09:28:15.016 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../103-python3-olefile_0.45.1-1_all.deb ... 2021-04-04T09:28:15.061 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-04-04T09:28:15.305 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-olefile. 2021-04-04T09:28:15.322 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../103-python3-olefile_0.45.1-1_all.deb ... 2021-04-04T09:28:15.363 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-04-04T09:28:15.388 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pastescript. 2021-04-04T09:28:15.403 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../104-python3-pastescript_2.0.2-2_all.deb ... 2021-04-04T09:28:15.455 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-04-04T09:28:15.675 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pastescript. 2021-04-04T09:28:15.688 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../104-python3-pastescript_2.0.2-2_all.deb ... 2021-04-04T09:28:15.732 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-04-04T09:28:15.775 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pil:amd64. 2021-04-04T09:28:15.791 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../105-python3-pil_5.1.0-1ubuntu0.5_amd64.deb ... 2021-04-04T09:28:15.834 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.5) ... 2021-04-04T09:28:16.053 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pil:amd64. 2021-04-04T09:28:16.075 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../105-python3-pil_5.1.0-1ubuntu0.5_amd64.deb ... 2021-04-04T09:28:16.111 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.5) ... 2021-04-04T09:28:16.204 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pluggy. 2021-04-04T09:28:16.219 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../106-python3-pluggy_0.6.0-1_all.deb ... 2021-04-04T09:28:16.262 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-04-04T09:28:16.473 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pluggy. 2021-04-04T09:28:16.490 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../106-python3-pluggy_0.6.0-1_all.deb ... 2021-04-04T09:28:16.531 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-04-04T09:28:16.548 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-py. 2021-04-04T09:28:16.562 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../107-python3-py_1.5.2-1_all.deb ... 2021-04-04T09:28:16.606 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-py (1.5.2-1) ... 2021-04-04T09:28:16.835 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-py. 2021-04-04T09:28:16.849 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../107-python3-py_1.5.2-1_all.deb ... 2021-04-04T09:28:16.893 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-py (1.5.2-1) ... 2021-04-04T09:28:16.956 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyinotify. 2021-04-04T09:28:16.967 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1_all.deb ... 2021-04-04T09:28:17.010 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-04-04T09:28:17.222 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pyinotify. 2021-04-04T09:28:17.238 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1_all.deb ... 2021-04-04T09:28:17.280 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-04-04T09:28:17.329 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pytest. 2021-04-04T09:28:17.344 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../109-python3-pytest_3.3.2-2_all.deb ... 2021-04-04T09:28:17.396 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-04-04T09:28:17.609 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-pytest. 2021-04-04T09:28:17.622 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../109-python3-pytest_3.3.2-2_all.deb ... 2021-04-04T09:28:17.667 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-04-04T09:28:17.733 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-simplejson. 2021-04-04T09:28:17.751 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../110-python3-simplejson_3.13.2-1_amd64.deb ... 2021-04-04T09:28:17.800 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-04-04T09:28:18.037 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-simplejson. 2021-04-04T09:28:18.052 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../110-python3-simplejson_3.13.2-1_amd64.deb ... 2021-04-04T09:28:18.095 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-04-04T09:28:18.145 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-tk:amd64. 2021-04-04T09:28:18.161 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../111-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-04-04T09:28:18.202 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-04-04T09:28:18.440 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package python3-tk:amd64. 2021-04-04T09:28:18.454 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../111-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-04-04T09:28:18.499 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-04-04T09:28:18.673 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package radosgw. 2021-04-04T09:28:18.690 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../112-radosgw_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:28:18.732 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking radosgw (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:18.936 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package radosgw. 2021-04-04T09:28:18.952 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../112-radosgw_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:28:18.995 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking radosgw (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:19.810 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package rbd-fuse. 2021-04-04T09:28:19.826 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../113-rbd-fuse_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:28:19.885 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking rbd-fuse (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:20.057 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package rbd-fuse. 2021-04-04T09:28:20.072 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../113-rbd-fuse_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:28:20.123 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking rbd-fuse (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:20.229 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package rbd-mirror. 2021-04-04T09:28:20.245 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../114-rbd-mirror_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:28:20.287 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking rbd-mirror (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:20.468 INFO:teuthology.orchestra.run.smithi122.stdout:Selecting previously unselected package rbd-mirror. 2021-04-04T09:28:20.483 INFO:teuthology.orchestra.run.smithi122.stdout:Preparing to unpack .../114-rbd-mirror_16.2.0-22-gf4fb4141-1bionic_amd64.deb ... 2021-04-04T09:28:20.543 INFO:teuthology.orchestra.run.smithi122.stdout:Unpacking rbd-mirror (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:20.899 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-04-04T09:28:21.165 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-04-04T09:28:21.218 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-04-04T09:28:21.483 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-04-04T09:28:21.570 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-04-04T09:28:21.704 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-04-04T09:28:21.859 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-04-04T09:28:21.986 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-04-04T09:28:22.323 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.5) ... 2021-04-04T09:28:22.589 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.5) ... 2021-04-04T09:28:22.708 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-04-04T09:28:22.973 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-04-04T09:28:23.001 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-webencodings (0.5-2) ... 2021-04-04T09:28:23.266 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-webencodings (0.5-2) ... 2021-04-04T09:28:23.294 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-attr (17.4.0-2) ... 2021-04-04T09:28:23.495 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-04-04T09:28:23.577 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python-attr (17.4.0-2) ... 2021-04-04T09:28:23.621 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-py (1.5.2-1) ... 2021-04-04T09:28:23.777 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-04-04T09:28:23.903 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-py (1.5.2-1) ... 2021-04-04T09:28:23.948 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-04-04T09:28:24.238 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-04-04T09:28:24.281 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-04-04T09:28:24.587 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-04-04T09:28:24.599 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-04-04T09:28:24.894 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-ceph-argparse (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:24.900 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-04-04T09:28:25.193 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-ceph-argparse (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:25.203 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-04-04T09:28:25.502 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-04-04T09:28:25.512 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-04-04T09:28:25.647 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-04-04T09:28:25.812 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-04-04T09:28:25.938 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-04-04T09:28:26.575 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-04-04T09:28:26.866 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-04-04T09:28:26.901 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-04-04T09:28:27.185 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-04-04T09:28:27.202 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-py (1.5.2-1) ... 2021-04-04T09:28:27.446 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-04-04T09:28:27.877 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python-py (1.5.2-1) ... 2021-04-04T09:28:28.055 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-04-04T09:28:28.106 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-04-04T09:28:28.154 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-04-04T09:28:28.374 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-04-04T09:28:28.449 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-04-04T09:28:28.500 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-04-04T09:28:28.538 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-04-04T09:28:28.693 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-04-04T09:28:28.759 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-04-04T09:28:28.885 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-04-04T09:28:28.985 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-04-04T09:28:29.086 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-04-04T09:28:29.278 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-04-04T09:28:29.378 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-04-04T09:28:29.571 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-04-04T09:28:29.696 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-04-04T09:28:29.939 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-04-04T09:28:30.014 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-04-04T09:28:30.265 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-04-04T09:28:30.390 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-04-04T09:28:30.550 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up cephadm (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:30.725 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-04-04T09:28:31.026 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up cephadm (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:31.242 INFO:teuthology.orchestra.run.smithi078.stdout:Adding system user cephadm....done 2021-04-04T09:28:31.415 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-04-04T09:28:31.668 INFO:teuthology.orchestra.run.smithi122.stdout:Adding system user cephadm....done 2021-04-04T09:28:31.708 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-04-04T09:28:31.834 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-04-04T09:28:31.841 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-04-04T09:28:32.010 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-04-04T09:28:32.142 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-04-04T09:28:32.268 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-04-04T09:28:32.361 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-04-04T09:28:32.411 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-04-04T09:28:32.654 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-04-04T09:28:32.746 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-04-04T09:28:32.780 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-04-04T09:28:33.046 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-04-04T09:28:33.107 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-ceph-common (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:33.172 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-04-04T09:28:33.425 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-monotonic (1.1-2) ... 2021-04-04T09:28:33.507 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-ceph-common (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:33.726 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-04-04T09:28:33.817 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-monotonic (1.1-2) ... 2021-04-04T09:28:33.852 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-04-04T09:28:34.118 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-04-04T09:28:34.212 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-04-04T09:28:34.260 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-04-04T09:28:34.571 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-04-04T09:28:34.661 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-04-04T09:28:34.696 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-04-04T09:28:34.703 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-04-04T09:28:34.934 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-04-04T09:28:35.063 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-04-04T09:28:35.152 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-04-04T09:28:35.184 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-04-04T09:28:35.235 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-04-04T09:28:35.424 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-04-04T09:28:35.545 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-04-04T09:28:35.725 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-04-04T09:28:35.887 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-attr (17.4.0-2) ... 2021-04-04T09:28:36.026 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-04-04T09:28:36.205 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-04-04T09:28:36.343 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-attr (17.4.0-2) ... 2021-04-04T09:28:36.424 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libjaeger (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:36.548 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-04-04T09:28:36.643 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-04-04T09:28:36.820 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libjaeger (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:36.946 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-04-04T09:28:36.958 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-04-04T09:28:37.048 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-04-04T09:28:37.302 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up liboath0 (2.6.1-1) ... 2021-04-04T09:28:37.373 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-04-04T09:28:37.463 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-04-04T09:28:37.478 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-04-04T09:28:37.612 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-04-04T09:28:37.725 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up liboath0 (2.6.1-1) ... 2021-04-04T09:28:37.896 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-04-04T09:28:37.909 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-04-04T09:28:38.019 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-04-04T09:28:38.023 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-04-04T09:28:38.245 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-04-04T09:28:38.345 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-04-04T09:28:38.416 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-04-04T09:28:38.542 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-04-04T09:28:38.722 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-04-04T09:28:38.843 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-tz (2018.3-2) ... 2021-04-04T09:28:38.848 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-04-04T09:28:39.127 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-04-04T09:28:39.157 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-tz (2018.3-2) ... 2021-04-04T09:28:39.428 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-04-04T09:28:39.459 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-04-04T09:28:39.713 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-joblib (0.11-1) ... 2021-04-04T09:28:39.759 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-04-04T09:28:40.069 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-joblib (0.11-1) ... 2021-04-04T09:28:40.081 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-routes (2.4.1-1) ... 2021-04-04T09:28:40.390 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-04-04T09:28:40.412 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-routes (2.4.1-1) ... 2021-04-04T09:28:40.691 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-04-04T09:28:40.696 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-04-04T09:28:41.014 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-04-04T09:28:41.017 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-04-04T09:28:41.332 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-04-04T09:28:41.377 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-04-04T09:28:41.653 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up jq (1.5+dfsg-2) ... 2021-04-04T09:28:41.700 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-04-04T09:28:41.754 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-04-04T09:28:42.001 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up jq (1.5+dfsg-2) ... 2021-04-04T09:28:42.119 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-04-04T09:28:42.901 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.4) ... 2021-04-04T09:28:43.169 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-04-04T09:28:43.258 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.4) ... 2021-04-04T09:28:43.470 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-04-04T09:28:43.571 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-04-04T09:28:43.584 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-04-04T09:28:43.927 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-04-04T09:28:43.953 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-pytest (3.3.2-2) ... 2021-04-04T09:28:44.036 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-04-04T09:28:44.206 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-04-04T09:28:44.332 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-04-04T09:28:44.387 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python-pytest (3.3.2-2) ... 2021-04-04T09:28:44.467 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librados2 (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:44.593 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-04-04T09:28:44.630 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-04-04T09:28:44.756 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-04-04T09:28:44.882 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librados2 (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:44.885 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libcephfs2 (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:44.986 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libcephfs-dev (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:45.008 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-04-04T09:28:45.087 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rados (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:45.196 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-04-04T09:28:45.317 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libcephfs2 (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:45.447 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libcephfs-dev (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:45.509 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-04-04T09:28:45.573 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-rados (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:45.670 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-04-04T09:28:45.807 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-04-04T09:28:45.962 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-04-04T09:28:46.226 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-04-04T09:28:46.239 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-04-04T09:28:46.650 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-04-04T09:28:47.054 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-04-04T09:28:47.486 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-04-04T09:28:47.883 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-04-04T09:28:48.193 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libradosstriper1 (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:48.291 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-04-04T09:28:48.310 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cephfs (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:48.578 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libsqlite3-mod-ceph (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:48.583 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libradosstriper1 (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:48.679 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librgw2 (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:48.709 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-cephfs (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:48.779 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-fuse (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:48.989 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-04-04T09:28:48.990 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-04-04T09:28:49.010 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up libsqlite3-mod-ceph (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:49.128 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librgw2 (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:49.228 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-fuse (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:49.273 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librbd1 (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:49.398 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-04-04T09:28:49.435 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-04-04T09:28:49.436 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-04-04T09:28:49.488 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-04-04T09:28:49.528 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-04-04T09:28:49.731 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up librbd1 (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:49.827 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-04-04T09:28:49.856 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-04-04T09:28:49.963 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-04-04T09:28:50.006 INFO:teuthology.orchestra.run.smithi122.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-04-04T09:28:50.309 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-04-04T09:28:50.471 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-modules-core (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:50.571 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-04-04T09:28:50.898 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rgw (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:51.007 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-modules-core (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:51.023 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up rbd-fuse (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:51.105 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-04-04T09:28:51.149 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rbd (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:51.275 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-common (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:51.397 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-rgw (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:51.524 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up rbd-fuse (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:51.649 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up python3-rbd (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:51.760 INFO:teuthology.orchestra.run.smithi078.stdout:Adding group ceph....done 2021-04-04T09:28:51.775 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-common (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:52.162 INFO:teuthology.orchestra.run.smithi078.stdout:Adding system user ceph....done 2021-04-04T09:28:52.270 INFO:teuthology.orchestra.run.smithi122.stdout:Adding group ceph....done 2021-04-04T09:28:52.346 INFO:teuthology.orchestra.run.smithi078.stdout:Setting system user ceph properties....done 2021-04-04T09:28:52.369 INFO:teuthology.orchestra.run.smithi078.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2021-04-04T09:28:52.505 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-04-04T09:28:52.621 INFO:teuthology.orchestra.run.smithi122.stdout:Adding system user ceph....done 2021-04-04T09:28:52.733 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-04-04T09:28:52.841 INFO:teuthology.orchestra.run.smithi122.stdout:Setting system user ceph properties....done 2021-04-04T09:28:52.862 INFO:teuthology.orchestra.run.smithi122.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2021-04-04T09:28:53.002 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-04-04T09:28:53.018 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up radosgw (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:53.216 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-04-04T09:28:53.283 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-04-04T09:28:53.284 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-04-04T09:28:53.511 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up radosgw (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:53.704 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up rbd-mirror (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:53.782 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-04-04T09:28:53.783 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-04-04T09:28:53.907 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2021-04-04T09:28:53.907 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2021-04-04T09:28:54.172 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-test (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:54.179 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up rbd-mirror (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:54.303 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-base (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:54.379 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2021-04-04T09:28:54.379 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2021-04-04T09:28:54.639 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-test (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:54.656 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-04-04T09:28:54.757 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-base (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:54.958 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:55.093 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-04-04T09:28:55.177 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-04-04T09:28:55.177 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-04-04T09:28:55.375 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:55.493 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-diskprediction-local (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:55.599 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-04-04T09:28:55.599 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-04-04T09:28:55.635 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-osd (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:55.884 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-diskprediction-local (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:56.027 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-osd (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:56.063 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-04-04T09:28:56.064 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-04-04T09:28:56.354 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-cephadm (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:56.463 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-rook (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:56.480 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-04-04T09:28:56.480 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-04-04T09:28:56.589 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mds (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:56.771 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-cephadm (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:56.811 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-04-04T09:28:56.811 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-04-04T09:28:56.897 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-rook (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:57.023 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mds (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:57.099 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mon (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:57.250 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-04-04T09:28:57.250 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-04-04T09:28:57.313 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-04-04T09:28:57.314 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-04-04T09:28:57.549 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mon (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:57.609 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-k8sevents (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:57.734 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-dashboard (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:57.771 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-04-04T09:28:57.772 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-04-04T09:28:57.877 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:57.987 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-04-04T09:28:58.075 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-k8sevents (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:58.196 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-04-04T09:28:58.744 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph-mgr-dashboard (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:58.895 INFO:teuthology.orchestra.run.smithi122.stdout:Setting up ceph (16.2.0-22-gf4fb4141-1bionic) ... 2021-04-04T09:28:59.005 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-04-04T09:28:59.223 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-04-04T09:29:00.391 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-04-04T09:29:00.434 INFO:teuthology.orchestra.run.smithi078.stdout:ureadahead will be reprofiled on next reboot 2021-04-04T09:29:00.500 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-04-04T09:29:00.948 INFO:teuthology.orchestra.run.smithi078.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-04-04T09:29:00.952 DEBUG:teuthology.parallel:result is None 2021-04-04T09:29:01.450 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-04-04T09:29:01.492 INFO:teuthology.orchestra.run.smithi122.stdout:ureadahead will be reprofiled on next reboot 2021-04-04T09:29:01.551 INFO:teuthology.orchestra.run.smithi122.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-04-04T09:29:02.002 INFO:teuthology.orchestra.run.smithi122.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-04-04T09:29:02.004 DEBUG:teuthology.parallel:result is None 2021-04-04T09:29:02.005 DEBUG:teuthology.orchestra.run.smithi078:> sudo lsb_release -is 2021-04-04T09:29:02.062 INFO:teuthology.orchestra.run.smithi078.stdout:Ubuntu 2021-04-04T09:29:02.067 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-04-04T09:29:02.068 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=f4fb41419a42bee43a2069637ef17064073bf30b 2021-04-04T09:29:02.250 DEBUG:teuthology.orchestra.run.smithi078:> dpkg-query -W -f '${Version}' ceph 2021-04-04T09:29:02.275 INFO:teuthology.orchestra.run.smithi078.stdout:16.2.0-22-gf4fb4141-1bionic 2021-04-04T09:29:02.276 INFO:teuthology.packaging:The installed version of ceph is 16.2.0-22-gf4fb4141-1bionic 2021-04-04T09:29:02.277 INFO:teuthology.task.install:The correct ceph version 16.2.0-22-gf4fb4141-1bionic is installed. 2021-04-04T09:29:02.277 DEBUG:teuthology.orchestra.run.smithi122:> sudo lsb_release -is 2021-04-04T09:29:02.331 INFO:teuthology.orchestra.run.smithi122.stdout:Ubuntu 2021-04-04T09:29:02.337 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-04-04T09:29:02.338 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=f4fb41419a42bee43a2069637ef17064073bf30b 2021-04-04T09:29:02.472 DEBUG:teuthology.orchestra.run.smithi122:> dpkg-query -W -f '${Version}' ceph 2021-04-04T09:29:02.500 INFO:teuthology.orchestra.run.smithi122.stdout:16.2.0-22-gf4fb4141-1bionic 2021-04-04T09:29:02.500 INFO:teuthology.packaging:The installed version of ceph is 16.2.0-22-gf4fb4141-1bionic 2021-04-04T09:29:02.501 INFO:teuthology.task.install:The correct ceph version 16.2.0-22-gf4fb4141-1bionic is installed. 2021-04-04T09:29:02.501 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-04-04T09:29:02.503 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:02.504 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-04-04T09:29:02.520 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:02.521 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-04-04T09:29:02.554 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-04-04T09:29:02.556 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:02.556 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2021-04-04T09:29:02.577 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-04-04T09:29:02.636 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:02.636 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2021-04-04T09:29:02.652 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-04-04T09:29:02.708 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-04-04T09:29:02.708 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:02.709 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2021-04-04T09:29:02.725 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-04-04T09:29:02.780 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:02.780 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2021-04-04T09:29:02.796 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-04-04T09:29:02.855 INFO:teuthology.run_tasks:Running task ceph... 2021-04-04T09:29:02.952 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-04-04T09:29:02.953 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /var/log/ceph 2021-04-04T09:29:02.956 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /var/log/ceph 2021-04-04T09:29:02.971 INFO:tasks.ceph:Disabling ceph logrotate... 2021-04-04T09:29:02.971 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/logrotate.d/ceph 2021-04-04T09:29:03.015 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/logrotate.d/ceph 2021-04-04T09:29:03.031 INFO:tasks.ceph:Creating extra log directories... 2021-04-04T09:29:03.031 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-04-04T09:29:03.071 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-04-04T09:29:03.105 INFO:tasks.ceph:Creating ceph cluster cluster1... 2021-04-04T09:29:03.105 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-04-04T09:29:03.106 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2021-04-04_06:07:03-rbd-pacific-distro-basic-smithi/6019607', 'branch': 'pacific', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-policy-simple}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '6019607', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror_journal.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2021-04-04_06:07:03-rbd-pacific-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '18.04', 'overrides': {'admin_socket': {'branch': 'pacific'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b'}}, 'workunit': {'branch': 'pacific', 'sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 80, 'repo': 'https://github.com/ceph/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b', 'sleep_before_teardown': 0, 'suite': 'rbd', 'suite_branch': 'pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_f4fb41419a42bee43a2069637ef17064073bf30b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b', 'targets': {'smithi078.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCox+hBSbqhZEOOs8jJK9PcXf99OuJixxSDbvBBJ8g0Jqe3aOnAMO+buNziqMRwbKJXzwgNnWt9t7jtlfJE+H+CfWSV1mTaaI4R3SGDIdVGXHuu38Y7B1XVImZeAvGYrlvVuquCp6oLBt4bucRosTI7NuXdLkm6f6DCOQJkhV09jum2gvIT0VtQrSEAFKIDU129Cr1q9zKd+7mHZwHM2KrT0974/I2dvr5qXy+6MKfDRd5P8wwKoVtlinNbVpPZ6NVyIKm5aOa+QOE7W2jIV/ObgS34+grwZ3kDeE5GvqcI7tZErKB5M5zat6zk125M+fyQ/xSBbE6I70tQ3kqoSDC7', 'smithi122.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0B/RVSuPh8DWhZMw8FntzsniAdnj1e24NvMQ7NpKZhgVshkWgPW0KytLhlnNC+PusZl6HqGhurZeByzBqf8oZwNNJa03FA9WRMHEURT6LJnP54V4HePaqUTn8mv8dALIvAyKSBiRLhEc99GOpKnJhwOqnLi/QimZerlOBSQbyNgtCdsZnKCpmlHpXI6NWwK1VzlblAhCLMnGdlLzCDQW5NJ53DfbIZNCFUX3r7Xn2bHzJyYCKVkIVIMhK4h0f0X+HxlkJ0zHzPrutH8Yzl96PUPn7TyttNXI3jhPpaaFXBGxk9OvHSxBvb87eIeWbaG8fz/xuz6DxcP6MazsCcwcn'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_journal.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '85d61eae4759f46ce21e9a37cd816a7a1a66c9d5', 'timestamp': '2021-04-04_06:07:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.13277'} 2021-04-04T09:29:03.107 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2021-04-04T09:29:03.135 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2021-04-04T09:29:03.147 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/run/ceph 2021-04-04T09:29:03.187 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/run/ceph 2021-04-04T09:29:03.204 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:03.205 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2021-04-04T09:29:03.247 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-04-04T09:29:03.248 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-04-04T09:29:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-04-04T09:29:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-04-04T09:29:03.296 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 341 Links: 1 2021-04-04T09:29:03.297 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-04-04T09:29:03.297 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2021-04-04 09:26:25.532897230 +0000 2021-04-04T09:29:03.297 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2021-04-04 09:26:25.528897230 +0000 2021-04-04T09:29:03.298 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2021-04-04 09:26:25.528897230 +0000 2021-04-04T09:29:03.298 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2021-04-04T09:29:03.305 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2021-04-04T09:29:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2021-04-04T09:29:03.306 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000387138 s, 1.3 MB/s 2021-04-04T09:29:03.310 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-04-04T09:29:03.359 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-04-04T09:29:03.360 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-04-04T09:29:03.360 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 348 Links: 1 2021-04-04T09:29:03.360 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-04-04T09:29:03.360 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2021-04-04 09:26:25.880897226 +0000 2021-04-04T09:29:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2021-04-04 09:26:25.760897228 +0000 2021-04-04T09:29:03.361 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2021-04-04 09:26:25.760897228 +0000 2021-04-04T09:29:03.361 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2021-04-04T09:29:03.369 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2021-04-04T09:29:03.369 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2021-04-04T09:29:03.370 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000320613 s, 1.6 MB/s 2021-04-04T09:29:03.373 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-04-04T09:29:03.423 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-04-04T09:29:03.424 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-04-04T09:29:03.424 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 354 Links: 1 2021-04-04T09:29:03.424 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-04-04T09:29:03.424 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2021-04-04 09:26:26.120897224 +0000 2021-04-04T09:29:03.425 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2021-04-04 09:26:26.024897225 +0000 2021-04-04T09:29:03.425 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2021-04-04 09:26:26.024897225 +0000 2021-04-04T09:29:03.425 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2021-04-04T09:29:03.433 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2021-04-04T09:29:03.433 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2021-04-04T09:29:03.433 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000382507 s, 1.3 MB/s 2021-04-04T09:29:03.437 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-04-04T09:29:03.487 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-04-04T09:29:03.488 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-04-04T09:29:03.488 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 360 Links: 1 2021-04-04T09:29:03.488 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-04-04T09:29:03.488 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2021-04-04 09:26:26.352897221 +0000 2021-04-04T09:29:03.489 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2021-04-04 09:26:26.260897222 +0000 2021-04-04T09:29:03.489 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2021-04-04 09:26:26.260897222 +0000 2021-04-04T09:29:03.489 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2021-04-04T09:29:03.497 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2021-04-04T09:29:03.498 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2021-04-04T09:29:03.498 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000400114 s, 1.3 MB/s 2021-04-04T09:29:03.502 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2021-04-04T09:29:03.503 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-04-04T09:29:03.503 INFO:tasks.ceph:Generating config... 2021-04-04T09:29:03.507 INFO:tasks.ceph:[client] debug rbd = 20 2021-04-04T09:29:03.508 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2021-04-04T09:29:03.508 INFO:tasks.ceph:[client] log to stderr = False 2021-04-04T09:29:03.509 INFO:tasks.ceph:[client] rbd default features = 125 2021-04-04T09:29:03.510 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2021-04-04T09:29:03.510 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:03.511 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:03.511 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:03.512 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:03.512 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:03.513 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:03.514 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:03.515 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:03.521 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:03.522 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:03.523 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:03.523 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:03.524 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:03.525 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:03.525 INFO:tasks.ceph:[global] mon client directed command retry = 5 2021-04-04T09:29:03.526 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2021-04-04T09:29:03.526 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-04-04T09:29:03.527 INFO:tasks.ceph:[mgr] debug ms = 1 2021-04-04T09:29:03.527 INFO:tasks.ceph:[mon] debug mon = 20 2021-04-04T09:29:03.528 INFO:tasks.ceph:[mon] debug ms = 1 2021-04-04T09:29:03.528 INFO:tasks.ceph:[mon] debug paxos = 20 2021-04-04T09:29:03.528 INFO:tasks.ceph:[osd] debug ms = 1 2021-04-04T09:29:03.529 INFO:tasks.ceph:[osd] debug osd = 20 2021-04-04T09:29:03.529 INFO:tasks.ceph:[osd] osd objectstore = filestore 2021-04-04T09:29:03.529 INFO:tasks.ceph:[osd] osd sloppy crc = True 2021-04-04T09:29:03.530 INFO:tasks.ceph:Setting up cluster1.mon.a... 2021-04-04T09:29:03.530 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2021-04-04T09:29:03.576 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/cluster1.keyring 2021-04-04T09:29:03.579 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2021-04-04T09:29:03.655 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2021-04-04T09:29:03.710 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.78')] 2021-04-04T09:29:03.712 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.78', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2021-04-04T09:29:03.713 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:03.714 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-04-04T09:29:03.760 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.78 --print /home/ubuntu/cephtest/cluster1.monmap 2021-04-04T09:29:03.831 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2021-04-04T09:29:03.832 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: generated fsid 7fae7b04-7e84-4b67-8401-2f7942dda448 2021-04-04T09:29:03.832 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 0 2021-04-04T09:29:03.832 INFO:teuthology.orchestra.run.smithi078.stdout:fsid 7fae7b04-7e84-4b67-8401-2f7942dda448 2021-04-04T09:29:03.833 INFO:teuthology.orchestra.run.smithi078.stdout:last_changed 2021-04-04T09:29:03.830952+0000 2021-04-04T09:29:03.833 INFO:teuthology.orchestra.run.smithi078.stdout:created 2021-04-04T09:29:03.830952+0000 2021-04-04T09:29:03.834 INFO:teuthology.orchestra.run.smithi078.stdout:min_mon_release 0 (unknown) 2021-04-04T09:29:03.834 INFO:teuthology.orchestra.run.smithi078.stdout:election_strategy: 1 2021-04-04T09:29:03.834 INFO:teuthology.orchestra.run.smithi078.stdout:0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2021-04-04T09:29:03.835 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2021-04-04T09:29:03.838 DEBUG:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-04-04T09:29:03.887 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 7fae7b04-7e84-4b67-8401-2f7942dda448... 2021-04-04T09:29:03.889 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2021-04-04T09:29:03.931 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2021-04-04T09:29:03.963 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2021-04-04T09:29:03.964 INFO:teuthology.orchestra.run.smithi078.stdout: chdir = "" 2021-04-04T09:29:03.964 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-04-04T09:29:03.964 INFO:teuthology.orchestra.run.smithi078.stdout: auth supported = cephx 2021-04-04T09:29:03.964 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.965 INFO:teuthology.orchestra.run.smithi078.stdout: filestore xattr use omap = true 2021-04-04T09:29:03.965 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.965 INFO:teuthology.orchestra.run.smithi078.stdout: mon clock drift allowed = 1.000 2021-04-04T09:29:03.965 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.966 INFO:teuthology.orchestra.run.smithi078.stdout: osd crush chooseleaf type = 0 2021-04-04T09:29:03.966 INFO:teuthology.orchestra.run.smithi078.stdout: auth debug = true 2021-04-04T09:29:03.966 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.966 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on old message = true 2021-04-04T09:29:03.966 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on bug = true 2021-04-04T09:29:03.967 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.967 INFO:teuthology.orchestra.run.smithi078.stdout: mon max pg per osd = 10000 # >= luminous 2021-04-04T09:29:03.967 INFO:teuthology.orchestra.run.smithi078.stdout: mon pg warn max object skew = 0 2021-04-04T09:29:03.967 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.968 INFO:teuthology.orchestra.run.smithi078.stdout: # disable pg_autoscaler by default for new pools 2021-04-04T09:29:03.968 INFO:teuthology.orchestra.run.smithi078.stdout: osd_pool_default_pg_autoscale_mode = off 2021-04-04T09:29:03.968 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.968 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default size = 2 2021-04-04T09:29:03.968 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.969 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow primary affinity = true 2021-04-04T09:29:03.969 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow pg remap = true 2021-04-04T09:29:03.969 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on legacy crush tunables = false 2021-04-04T09:29:03.969 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on crush straw calc version zero = false 2021-04-04T09:29:03.970 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on no sortbitwise = false 2021-04-04T09:29:03.970 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on osd down out interval zero = false 2021-04-04T09:29:03.970 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on too few osds = false 2021-04-04T09:29:03.970 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-04-04T09:29:03.971 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_no_redundancy = false 2021-04-04T09:29:03.971 INFO:teuthology.orchestra.run.smithi078.stdout: mon_allow_pool_size_one = true 2021-04-04T09:29:03.971 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.971 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-04-04T09:29:03.972 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.972 INFO:teuthology.orchestra.run.smithi078.stdout: osd default data pool replay window = 5 2021-04-04T09:29:03.972 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.973 INFO:teuthology.orchestra.run.smithi078.stdout: mon allow pool delete = true 2021-04-04T09:29:03.973 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.973 INFO:teuthology.orchestra.run.smithi078.stdout: mon cluster log file level = debug 2021-04-04T09:29:03.973 INFO:teuthology.orchestra.run.smithi078.stdout: debug asserts on shutdown = true 2021-04-04T09:29:03.974 INFO:teuthology.orchestra.run.smithi078.stdout: mon health detail to clog = false 2021-04-04T09:29:03.974 INFO:teuthology.orchestra.run.smithi078.stdout: mon host = 172.21.15.78 2021-04-04T09:29:03.975 INFO:teuthology.orchestra.run.smithi078.stdout: mon client directed command retry = 5 2021-04-04T09:29:03.975 INFO:teuthology.orchestra.run.smithi078.stdout: ms inject socket failures = 5000 2021-04-04T09:29:03.975 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = 7fae7b04-7e84-4b67-8401-2f7942dda448 2021-04-04T09:29:03.975 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.975 INFO:teuthology.orchestra.run.smithi078.stdout:[osd] 2021-04-04T09:29:03.976 INFO:teuthology.orchestra.run.smithi078.stdout: osd journal size = 100 2021-04-04T09:29:03.976 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.976 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub load threshold = 5.0 2021-04-04T09:29:03.976 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub max interval = 600 2021-04-04T09:29:03.977 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.977 INFO:teuthology.orchestra.run.smithi078.stdout: osd recover clone overlap = true 2021-04-04T09:29:03.977 INFO:teuthology.orchestra.run.smithi078.stdout: osd recovery max chunk = 1048576 2021-04-04T09:29:03.977 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.978 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug shutdown = true 2021-04-04T09:29:03.978 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug op order = true 2021-04-04T09:29:03.978 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify stray on activate = true 2021-04-04T09:29:03.978 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.978 INFO:teuthology.orchestra.run.smithi078.stdout: osd open classes on start = true 2021-04-04T09:29:03.979 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug pg log writeout = true 2021-04-04T09:29:03.979 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.979 INFO:teuthology.orchestra.run.smithi078.stdout: osd deep scrub update digest min age = 30 2021-04-04T09:29:03.979 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.980 INFO:teuthology.orchestra.run.smithi078.stdout: osd map max advance = 10 2021-04-04T09:29:03.980 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.980 INFO:teuthology.orchestra.run.smithi078.stdout: journal zero on create = true 2021-04-04T09:29:03.980 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.981 INFO:teuthology.orchestra.run.smithi078.stdout: filestore ondisk finisher threads = 3 2021-04-04T09:29:03.981 INFO:teuthology.orchestra.run.smithi078.stdout: filestore apply finisher threads = 3 2021-04-04T09:29:03.982 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.982 INFO:teuthology.orchestra.run.smithi078.stdout: bdev debug aio = true 2021-04-04T09:29:03.982 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug misdirected ops = true 2021-04-04T09:29:03.983 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2021-04-04T09:29:03.983 INFO:teuthology.orchestra.run.smithi078.stdout: debug osd = 20 2021-04-04T09:29:03.983 INFO:teuthology.orchestra.run.smithi078.stdout: osd objectstore = filestore 2021-04-04T09:29:03.984 INFO:teuthology.orchestra.run.smithi078.stdout: osd sloppy crc = True 2021-04-04T09:29:03.984 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.984 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr] 2021-04-04T09:29:03.984 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2021-04-04T09:29:03.985 INFO:teuthology.orchestra.run.smithi078.stdout: debug mgr = 20 2021-04-04T09:29:03.985 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2021-04-04T09:29:03.986 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2021-04-04T09:29:03.986 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2021-04-04T09:29:03.987 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2021-04-04T09:29:03.987 INFO:teuthology.orchestra.run.smithi078.stdout: mgr/telemetry/nag = false 2021-04-04T09:29:03.987 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.987 INFO:teuthology.orchestra.run.smithi078.stdout:[mon] 2021-04-04T09:29:03.988 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2021-04-04T09:29:03.988 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2021-04-04T09:29:03.988 INFO:teuthology.orchestra.run.smithi078.stdout: debug paxos = 20 2021-04-04T09:29:03.988 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2021-04-04T09:29:03.988 INFO:teuthology.orchestra.run.smithi078.stdout: mon data avail warn = 5 2021-04-04T09:29:03.989 INFO:teuthology.orchestra.run.smithi078.stdout: mon mgr mkfs grace = 240 2021-04-04T09:29:03.989 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2021-04-04T09:29:03.989 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd reporter subtree level = osd 2021-04-04T09:29:03.989 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd prime pg temp = true 2021-04-04T09:29:03.990 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2021-04-04T09:29:03.990 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:03.990 INFO:teuthology.orchestra.run.smithi078.stdout:[client] 2021-04-04T09:29:03.990 INFO:teuthology.orchestra.run.smithi078.stdout: rgw cache enabled = true 2021-04-04T09:29:03.990 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable ops log = true 2021-04-04T09:29:03.991 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable usage log = true 2021-04-04T09:29:03.991 INFO:teuthology.orchestra.run.smithi078.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-04-04T09:29:03.991 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-04-04T09:29:03.992 INFO:teuthology.orchestra.run.smithi078.stdout: debug rbd = 20 2021-04-04T09:29:03.992 INFO:teuthology.orchestra.run.smithi078.stdout: debug rbd_mirror = 15 2021-04-04T09:29:03.992 INFO:teuthology.orchestra.run.smithi078.stdout: log to stderr = False 2021-04-04T09:29:03.992 INFO:teuthology.orchestra.run.smithi078.stdout: rbd default features = 125 2021-04-04T09:29:03.993 INFO:teuthology.orchestra.run.smithi078.stdout: rbd mirror image policy type = simple 2021-04-04T09:29:03.993 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.a] 2021-04-04T09:29:03.993 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.0] 2021-04-04T09:29:03.993 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:03.994 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:03.994 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.1] 2021-04-04T09:29:03.994 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:03.994 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:03.995 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.2] 2021-04-04T09:29:03.995 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:03.995 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:03.995 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.3] 2021-04-04T09:29:03.996 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:03.996 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:03.996 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.4] 2021-04-04T09:29:03.996 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:03.996 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:03.997 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.5] 2021-04-04T09:29:03.997 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:03.998 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:03.998 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.6] 2021-04-04T09:29:03.999 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:03.999 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:04.002 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2021-04-04T09:29:04.002 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2021-04-04T09:29:04.002 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-04-04T09:29:04.003 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2021-04-04T09:29:04.003 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.003 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2021-04-04T09:29:04.003 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.004 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2021-04-04T09:29:04.004 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.004 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2021-04-04T09:29:04.005 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2021-04-04T09:29:04.005 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.005 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2021-04-04T09:29:04.005 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2021-04-04T09:29:04.005 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.006 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2021-04-04T09:29:04.006 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2021-04-04T09:29:04.006 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.006 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2021-04-04T09:29:04.007 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2021-04-04T09:29:04.007 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.007 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2021-04-04T09:29:04.007 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.008 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2021-04-04T09:29:04.008 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2021-04-04T09:29:04.008 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2021-04-04T09:29:04.008 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2021-04-04T09:29:04.008 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2021-04-04T09:29:04.009 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2021-04-04T09:29:04.009 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2021-04-04T09:29:04.009 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-04-04T09:29:04.009 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2021-04-04T09:29:04.010 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2021-04-04T09:29:04.010 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.010 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-04-04T09:29:04.010 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.011 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2021-04-04T09:29:04.011 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.011 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2021-04-04T09:29:04.012 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.012 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2021-04-04T09:29:04.013 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2021-04-04T09:29:04.013 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2021-04-04T09:29:04.013 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = 172.21.15.78 2021-04-04T09:29:04.013 INFO:teuthology.orchestra.run.smithi122.stdout: mon client directed command retry = 5 2021-04-04T09:29:04.014 INFO:teuthology.orchestra.run.smithi122.stdout: ms inject socket failures = 5000 2021-04-04T09:29:04.014 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = 7fae7b04-7e84-4b67-8401-2f7942dda448 2021-04-04T09:29:04.014 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.014 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2021-04-04T09:29:04.015 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2021-04-04T09:29:04.015 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.016 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2021-04-04T09:29:04.016 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2021-04-04T09:29:04.016 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.016 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2021-04-04T09:29:04.017 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2021-04-04T09:29:04.017 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.017 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2021-04-04T09:29:04.018 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2021-04-04T09:29:04.018 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2021-04-04T09:29:04.018 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.018 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2021-04-04T09:29:04.019 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2021-04-04T09:29:04.019 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.019 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2021-04-04T09:29:04.020 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.020 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2021-04-04T09:29:04.020 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.021 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2021-04-04T09:29:04.021 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.021 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2021-04-04T09:29:04.021 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2021-04-04T09:29:04.022 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.022 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2021-04-04T09:29:04.022 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2021-04-04T09:29:04.023 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2021-04-04T09:29:04.023 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2021-04-04T09:29:04.023 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = filestore 2021-04-04T09:29:04.024 INFO:teuthology.orchestra.run.smithi122.stdout: osd sloppy crc = True 2021-04-04T09:29:04.024 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.024 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2021-04-04T09:29:04.025 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2021-04-04T09:29:04.025 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2021-04-04T09:29:04.025 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2021-04-04T09:29:04.026 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2021-04-04T09:29:04.026 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2021-04-04T09:29:04.026 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2021-04-04T09:29:04.027 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2021-04-04T09:29:04.027 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.027 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2021-04-04T09:29:04.027 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2021-04-04T09:29:04.027 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2021-04-04T09:29:04.028 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2021-04-04T09:29:04.028 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2021-04-04T09:29:04.028 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2021-04-04T09:29:04.028 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2021-04-04T09:29:04.029 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2021-04-04T09:29:04.029 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2021-04-04T09:29:04.029 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2021-04-04T09:29:04.029 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2021-04-04T09:29:04.029 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:04.030 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2021-04-04T09:29:04.030 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2021-04-04T09:29:04.030 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2021-04-04T09:29:04.030 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2021-04-04T09:29:04.031 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-04-04T09:29:04.031 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-04-04T09:29:04.031 INFO:teuthology.orchestra.run.smithi122.stdout: debug rbd = 20 2021-04-04T09:29:04.031 INFO:teuthology.orchestra.run.smithi122.stdout: debug rbd_mirror = 15 2021-04-04T09:29:04.031 INFO:teuthology.orchestra.run.smithi122.stdout: log to stderr = False 2021-04-04T09:29:04.032 INFO:teuthology.orchestra.run.smithi122.stdout: rbd default features = 125 2021-04-04T09:29:04.032 INFO:teuthology.orchestra.run.smithi122.stdout: rbd mirror image policy type = simple 2021-04-04T09:29:04.032 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2021-04-04T09:29:04.033 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.0] 2021-04-04T09:29:04.033 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:04.033 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:04.033 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.1] 2021-04-04T09:29:04.034 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:04.034 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:04.034 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.2] 2021-04-04T09:29:04.035 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:04.035 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:04.035 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.3] 2021-04-04T09:29:04.035 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:04.036 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:04.036 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.4] 2021-04-04T09:29:04.036 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:04.036 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:04.036 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.5] 2021-04-04T09:29:04.037 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:04.037 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:04.037 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.6] 2021-04-04T09:29:04.037 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:04.037 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:04.038 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2021-04-04T09:29:04.038 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2021-04-04T09:29:04.090 INFO:tasks.ceph:Copying monmap to all nodes... 2021-04-04T09:29:04.091 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:04.092 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2021-04-04T09:29:04.099 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:04.099 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2021-04-04T09:29:04.148 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi078.front.sepia.ceph.com 2021-04-04T09:29:04.149 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:04.149 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/cluster1.keyring 2021-04-04T09:29:04.150 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2021-04-04T09:29:04.214 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:04.214 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2021-04-04T09:29:04.264 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi122.front.sepia.ceph.com 2021-04-04T09:29:04.265 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:04.266 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/cluster1.keyring 2021-04-04T09:29:04.267 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2021-04-04T09:29:04.293 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:04.294 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2021-04-04T09:29:04.343 INFO:tasks.ceph:Setting up mon nodes... 2021-04-04T09:29:04.343 INFO:tasks.ceph:Setting up mgr nodes... 2021-04-04T09:29:04.344 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2021-04-04T09:29:04.387 INFO:teuthology.orchestra.run.smithi078.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2021-04-04T09:29:04.389 INFO:tasks.ceph:Setting up mds nodes... 2021-04-04T09:29:04.390 INFO:tasks.ceph_client:Setting up client nodes... 2021-04-04T09:29:04.391 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2021-04-04T09:29:04.469 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/cluster1.client.0.keyring 2021-04-04T09:29:04.482 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2021-04-04T09:29:04.514 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2021-04-04T09:29:04.525 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2021-04-04T09:29:04.601 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2021-04-04T09:29:04.614 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2021-04-04T09:29:04.689 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2021-04-04T09:29:04.701 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2021-04-04T09:29:04.777 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2021-04-04T09:29:04.789 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2021-04-04T09:29:04.865 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2021-04-04T09:29:04.877 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2021-04-04T09:29:04.953 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2021-04-04T09:29:04.965 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2021-04-04T09:29:05.041 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2021-04-04T09:29:05.052 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2021-04-04T09:29:05.125 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2021-04-04T09:29:05.137 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-04-04T09:29:05.137 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-04-04T09:29:05.138 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2021-04-04T09:29:05.154 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2021-04-04T09:29:05.155 INFO:tasks.ceph:role: cluster1.osd.0 2021-04-04T09:29:05.155 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com 2021-04-04T09:29:05.157 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-04-04T09:29:05.692 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-04-04T09:29:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2021-04-04T09:29:05.692 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-04-04T09:29:05.693 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-04-04T09:29:05.693 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2021-04-04T09:29:05.694 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-04-04T09:29:05.694 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-04-04T09:29:05.694 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-04-04T09:29:05.695 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-04-04T09:29:05.696 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2021-04-04T09:29:05.697 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2021-04-04T09:29:05.769 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2021-04-04T09:29:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-04T09:29:05.781 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: /sbin/restorecon: command not found 2021-04-04T09:29:05.781 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2021-04-04T09:29:05.839 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2021-04-04T09:29:05.840 INFO:tasks.ceph:role: cluster1.osd.1 2021-04-04T09:29:05.840 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com 2021-04-04T09:29:05.840 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-04-04T09:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-04-04T09:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2021-04-04T09:29:06.357 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-04-04T09:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-04-04T09:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2021-04-04T09:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-04-04T09:29:06.358 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-04-04T09:29:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-04-04T09:29:06.359 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-04-04T09:29:06.360 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2021-04-04T09:29:06.360 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2021-04-04T09:29:06.424 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2021-04-04T09:29:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-04T09:29:06.436 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: /sbin/restorecon: command not found 2021-04-04T09:29:06.436 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2021-04-04T09:29:06.495 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2021-04-04T09:29:06.496 INFO:tasks.ceph:role: cluster1.osd.2 2021-04-04T09:29:06.497 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com 2021-04-04T09:29:06.497 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-04-04T09:29:07.014 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-04-04T09:29:07.015 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2021-04-04T09:29:07.015 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-04-04T09:29:07.016 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-04-04T09:29:07.016 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2021-04-04T09:29:07.016 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-04-04T09:29:07.017 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-04-04T09:29:07.017 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-04-04T09:29:07.017 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-04-04T09:29:07.019 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2021-04-04T09:29:07.020 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2021-04-04T09:29:07.072 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2021-04-04T09:29:07.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-04T09:29:07.087 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: /sbin/restorecon: command not found 2021-04-04T09:29:07.088 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2021-04-04T09:29:07.183 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:07.177+0000 7fc581aa8f00 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2021-04-04T09:29:07.184 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:07.177+0000 7fc581aa8f00 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2021-04-04T09:29:07.209 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:07.201+0000 7fc581aa8f00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:07.209 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:07.205+0000 7fc581aa8f00 -1 journal FileJournal::_open_file : zeroing journal 2021-04-04T09:29:07.449 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:07.445+0000 7fc581aa8f00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:07.452 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:07.445+0000 7fc581aa8f00 -1 journal do_read_entry(4096): bad header magic 2021-04-04T09:29:07.453 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:07.445+0000 7fc581aa8f00 -1 journal do_read_entry(4096): bad header magic 2021-04-04T09:29:07.469 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2021-04-04T09:29:07.480 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2021-04-04T09:29:07.558 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:07.553+0000 7fade35eef00 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2021-04-04T09:29:07.560 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:07.553+0000 7fade35eef00 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2021-04-04T09:29:07.596 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:07.589+0000 7fade35eef00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:07.597 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:07.589+0000 7fade35eef00 -1 journal FileJournal::_open_file : zeroing journal 2021-04-04T09:29:07.850 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:07.845+0000 7fade35eef00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:07.856 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:07.849+0000 7fade35eef00 -1 journal do_read_entry(4096): bad header magic 2021-04-04T09:29:07.856 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:07.849+0000 7fade35eef00 -1 journal do_read_entry(4096): bad header magic 2021-04-04T09:29:07.881 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2021-04-04T09:29:07.942 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2021-04-04T09:29:08.025 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:08.017+0000 7ff8cddc9f00 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2021-04-04T09:29:08.026 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:08.021+0000 7ff8cddc9f00 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2021-04-04T09:29:08.064 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:08.057+0000 7ff8cddc9f00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:08.065 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:08.057+0000 7ff8cddc9f00 -1 journal FileJournal::_open_file : zeroing journal 2021-04-04T09:29:08.325 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:08.321+0000 7ff8cddc9f00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:08.328 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:08.321+0000 7ff8cddc9f00 -1 journal do_read_entry(4096): bad header magic 2021-04-04T09:29:08.328 INFO:teuthology.orchestra.run.smithi078.stderr:2021-04-04T09:29:08.321+0000 7ff8cddc9f00 -1 journal do_read_entry(4096): bad header magic 2021-04-04T09:29:08.349 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2021-04-04T09:29:08.408 INFO:tasks.ceph:Reading keys from all nodes... 2021-04-04T09:29:08.409 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:08.409 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2021-04-04T09:29:08.462 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:08.463 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2021-04-04T09:29:08.520 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:08.521 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2021-04-04T09:29:08.580 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:08.581 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2021-04-04T09:29:08.636 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:08.637 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2021-04-04T09:29:08.684 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:08.684 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2021-04-04T09:29:08.692 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:08.692 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2021-04-04T09:29:08.739 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:08.739 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2021-04-04T09:29:08.788 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:08.789 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2021-04-04T09:29:08.835 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:08.835 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2021-04-04T09:29:08.883 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:08.883 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2021-04-04T09:29:08.932 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:08.932 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2021-04-04T09:29:08.979 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:08.979 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2021-04-04T09:29:09.027 INFO:tasks.ceph:Adding keys to all mons... 2021-04-04T09:29:09.027 DEBUG:teuthology.orchestra.run.smithi078:> sudo tee -a /etc/ceph/cluster1.keyring 2021-04-04T09:29:09.041 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr.x] 2021-04-04T09:29:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDghmlgWzMPFxAAEsHwhXKMw9TJCwfBVXaxGw== 2021-04-04T09:29:09.042 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.0] 2021-04-04T09:29:09.042 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDjhmlg5AnpChAA6nTwHps5stcAhqNW2v0UxA== 2021-04-04T09:29:09.043 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.1] 2021-04-04T09:29:09.043 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDjhmlg8TRGIRAAufIOl43JFoj12Sw6FkDL1w== 2021-04-04T09:29:09.044 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.2] 2021-04-04T09:29:09.044 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDkhmlgEVyFARAAuX2rIsOeWI0CCQSjRSqoww== 2021-04-04T09:29:09.044 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2021-04-04T09:29:09.045 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDghmlgz2f1GxAAgGlGr2AMJt3FQLEjDRgNaA== 2021-04-04T09:29:09.045 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror] 2021-04-04T09:29:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDghmlgbTLQHhAAqgyU+Gr4QlXHmjS1CGSf1Q== 2021-04-04T09:29:09.046 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.0] 2021-04-04T09:29:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDghmlgFu0BJBAAvGV1+IAgdseKbU7NU15PtQ== 2021-04-04T09:29:09.046 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.1] 2021-04-04T09:29:09.046 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDghmlgntNCKRAA7FAl89guKJHOrUESCp6nXA== 2021-04-04T09:29:09.047 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.2] 2021-04-04T09:29:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDghmlgtPqFLhAAXrAXdEz4Cs6501vr6oh5ug== 2021-04-04T09:29:09.047 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.3] 2021-04-04T09:29:09.047 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDghmlgyRe/MxAAHiaRm2k9D0QUlQmT7CkcCg== 2021-04-04T09:29:09.048 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.4] 2021-04-04T09:29:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDghmlgzAcBORAA5mPps51xzMJ5TT57CXcyyQ== 2021-04-04T09:29:09.048 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.5] 2021-04-04T09:29:09.048 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDhhmlgBo+iAhAAz/vkpjvTg7e91P9z6YhacA== 2021-04-04T09:29:09.048 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.6] 2021-04-04T09:29:09.049 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDhhmlgTnysBxAAA394PobLysBWIug9qjc98A== 2021-04-04T09:29:09.050 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-04-04T09:29:09.135 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-04T09:29:09.218 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-04T09:29:09.305 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-04T09:29:09.392 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:09.477 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:09.577 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:09.693 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:09.789 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:09.886 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:09.978 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:10.077 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:10.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:10.266 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-04-04T09:29:10.266 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2021-04-04T09:29:10.288 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2021-04-04T09:29:10.408 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2021-04-04T09:29:10.427 DEBUG:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2021-04-04T09:29:10.477 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2021-04-04T09:29:10.478 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2021-04-04T09:29:10.478 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2021-04-04T09:29:10.524 INFO:tasks.ceph.cluster1.mon.a:Started 2021-04-04T09:29:10.524 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2021-04-04T09:29:10.525 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2021-04-04T09:29:10.525 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2021-04-04T09:29:10.531 INFO:tasks.ceph.cluster1.mgr.x:Started 2021-04-04T09:29:10.531 INFO:tasks.ceph:Setting crush tunables to default 2021-04-04T09:29:10.532 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster cluster1 osd crush tunables default 2021-04-04T09:29:10.991 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2021-04-04T09:29:11.001 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2021-04-04T09:29:11.002 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:11.002 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2021-04-04T09:29:11.016 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:11.016 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2021-04-04T09:29:11.068 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:11.068 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2021-04-04T09:29:11.122 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster cluster1 osd new e6aeea0f-5441-4eab-8180-13833d6d9516 0 2021-04-04T09:29:11.387 INFO:teuthology.orchestra.run.smithi078.stdout:0 2021-04-04T09:29:11.400 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster cluster1 osd new 5d4eefd6-6f2f-4226-932d-2676c17b1796 1 2021-04-04T09:29:11.610 INFO:teuthology.orchestra.run.smithi078.stdout:1 2021-04-04T09:29:11.623 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster cluster1 osd new 538f2dff-c3d6-460a-942c-fb4e18295585 2 2021-04-04T09:29:11.827 INFO:teuthology.orchestra.run.smithi078.stdout:2 2021-04-04T09:29:11.840 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2021-04-04T09:29:11.840 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2021-04-04T09:29:11.843 INFO:tasks.ceph.cluster1.osd.0:Started 2021-04-04T09:29:11.843 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2021-04-04T09:29:11.843 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2021-04-04T09:29:11.845 INFO:tasks.ceph.cluster1.osd.1:Started 2021-04-04T09:29:11.846 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2021-04-04T09:29:11.846 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2021-04-04T09:29:11.848 INFO:tasks.ceph.cluster1.osd.2:Started 2021-04-04T09:29:11.849 INFO:tasks.ceph:Waiting for OSDs to come up 2021-04-04T09:29:11.903 INFO:tasks.ceph.cluster1.osd.0.smithi078.stderr:2021-04-04T09:29:11.897+0000 7f13eead4f00 -1 Falling back to public interface 2021-04-04T09:29:11.908 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T09:29:11.901+0000 7f43564d0f00 -1 Falling back to public interface 2021-04-04T09:29:11.913 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T09:29:11.909+0000 7f6a9f5c4f00 -1 Falling back to public interface 2021-04-04T09:29:11.917 INFO:tasks.ceph.cluster1.osd.0.smithi078.stderr:2021-04-04T09:29:11.913+0000 7f13eead4f00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:11.918 INFO:tasks.ceph.cluster1.osd.0.smithi078.stderr:2021-04-04T09:29:11.913+0000 7f13eead4f00 -1 journal do_read_entry(8192): bad header magic 2021-04-04T09:29:11.918 INFO:tasks.ceph.cluster1.osd.0.smithi078.stderr:2021-04-04T09:29:11.913+0000 7f13eead4f00 -1 journal do_read_entry(8192): bad header magic 2021-04-04T09:29:11.926 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T09:29:11.921+0000 7f43564d0f00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:11.927 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T09:29:11.921+0000 7f43564d0f00 -1 journal do_read_entry(8192): bad header magic 2021-04-04T09:29:11.927 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T09:29:11.921+0000 7f43564d0f00 -1 journal do_read_entry(8192): bad header magic 2021-04-04T09:29:11.932 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T09:29:11.925+0000 7f6a9f5c4f00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:11.933 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T09:29:11.925+0000 7f6a9f5c4f00 -1 journal do_read_entry(8192): bad header magic 2021-04-04T09:29:11.933 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T09:29:11.925+0000 7f6a9f5c4f00 -1 journal do_read_entry(8192): bad header magic 2021-04-04T09:29:11.935 INFO:tasks.ceph.cluster1.osd.0.smithi078.stderr:2021-04-04T09:29:11.929+0000 7f13eead4f00 -1 osd.0 0 log_to_monitors {default=true} 2021-04-04T09:29:11.945 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T09:29:11.941+0000 7f43564d0f00 -1 osd.1 0 log_to_monitors {default=true} 2021-04-04T09:29:11.956 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T09:29:11.949+0000 7f6a9f5c4f00 -1 osd.2 0 log_to_monitors {default=true} 2021-04-04T09:29:12.151 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2021-04-04T09:29:12.547 INFO:teuthology.misc.health.smithi078.stdout: 2021-04-04T09:29:12.548 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":5,"fsid":"7fae7b04-7e84-4b67-8401-2f7942dda448","created":"2021-04-04T09:29:10.649819+0000","modified":"2021-04-04T09:29:11.824921+0000","last_up_change":"0.000000","last_in_change":"2021-04-04T09:29:11.824921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"e6aeea0f-5441-4eab-8180-13833d6d9516","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5d4eefd6-6f2f-4226-932d-2676c17b1796","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"538f2dff-c3d6-460a-942c-fb4e18295585","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-04-04T09:29:12.559 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-04-04T09:29:13.835 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T09:29:13.829+0000 7f6a97a8e700 -1 osd.2 0 waiting for initial osdmap 2021-04-04T09:29:13.836 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T09:29:13.829+0000 7f434e199700 -1 osd.1 0 waiting for initial osdmap 2021-04-04T09:29:13.836 INFO:tasks.ceph.cluster1.osd.0.smithi078.stderr:2021-04-04T09:29:13.829+0000 7f13e679d700 -1 osd.0 0 waiting for initial osdmap 2021-04-04T09:29:18.863 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2021-04-04T09:29:19.162 INFO:teuthology.misc.health.smithi078.stdout: 2021-04-04T09:29:19.162 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":11,"fsid":"7fae7b04-7e84-4b67-8401-2f7942dda448","created":"2021-04-04T09:29:10.649819+0000","modified":"2021-04-04T09:29:17.853742+0000","last_up_change":"2021-04-04T09:29:14.833919+0000","last_in_change":"2021-04-04T09:29:11.824921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-04-04T09:29:15.544468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"e6aeea0f-5441-4eab-8180-13833d6d9516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":12840},{"type":"v1","addr":"172.21.15.78:6801","nonce":12840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":12840},{"type":"v1","addr":"172.21.15.78:6803","nonce":12840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":12840},{"type":"v1","addr":"172.21.15.78:6807","nonce":12840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":12840},{"type":"v1","addr":"172.21.15.78:6805","nonce":12840}]},"public_addr":"172.21.15.78:6801/12840","cluster_addr":"172.21.15.78:6803/12840","heartbeat_back_addr":"172.21.15.78:6807/12840","heartbeat_front_addr":"172.21.15.78:6805/12840","state":["exists","up"]},{"osd":1,"uuid":"5d4eefd6-6f2f-4226-932d-2676c17b1796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":12841},{"type":"v1","addr":"172.21.15.78:6809","nonce":12841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":12841},{"type":"v1","addr":"172.21.15.78:6811","nonce":12841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":12841},{"type":"v1","addr":"172.21.15.78:6815","nonce":12841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":12841},{"type":"v1","addr":"172.21.15.78:6813","nonce":12841}]},"public_addr":"172.21.15.78:6809/12841","cluster_addr":"172.21.15.78:6811/12841","heartbeat_back_addr":"172.21.15.78:6815/12841","heartbeat_front_addr":"172.21.15.78:6813/12841","state":["exists","up"]},{"osd":2,"uuid":"538f2dff-c3d6-460a-942c-fb4e18295585","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":12842},{"type":"v1","addr":"172.21.15.78:6817","nonce":12842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":12842},{"type":"v1","addr":"172.21.15.78:6819","nonce":12842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":12842},{"type":"v1","addr":"172.21.15.78:6823","nonce":12842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":12842},{"type":"v1","addr":"172.21.15.78:6821","nonce":12842}]},"public_addr":"172.21.15.78:6817/12842","cluster_addr":"172.21.15.78:6819/12842","heartbeat_back_addr":"172.21.15.78:6823/12842","heartbeat_front_addr":"172.21.15.78:6821/12842","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:12.947673+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:12.965073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:12.967267+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-04-04T09:29:19.175 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-04-04T09:29:19.176 INFO:tasks.ceph:Creating RBD pool 2021-04-04T09:29:19.176 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster cluster1 osd pool create rbd 8 2021-04-04T09:29:19.884 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'rbd' created 2021-04-04T09:29:19.897 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-04-04T09:29:20.890 INFO:teuthology.orchestra.run.smithi078.stderr:enabled application 'rbd' on pool 'rbd' 2021-04-04T09:29:20.903 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2021-04-04T09:29:20.904 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2021-04-04T09:29:20.904 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-04-04T09:29:21.197 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:21.198 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":13,"fsid":"7fae7b04-7e84-4b67-8401-2f7942dda448","created":"2021-04-04T09:29:10.649819+0000","modified":"2021-04-04T09:29:20.872161+0000","last_up_change":"2021-04-04T09:29:14.833919+0000","last_in_change":"2021-04-04T09:29:11.824921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-04-04T09:29:15.544468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-04-04T09:29:19.545788+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e6aeea0f-5441-4eab-8180-13833d6d9516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":12840},{"type":"v1","addr":"172.21.15.78:6801","nonce":12840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":12840},{"type":"v1","addr":"172.21.15.78:6803","nonce":12840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":12840},{"type":"v1","addr":"172.21.15.78:6807","nonce":12840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":12840},{"type":"v1","addr":"172.21.15.78:6805","nonce":12840}]},"public_addr":"172.21.15.78:6801/12840","cluster_addr":"172.21.15.78:6803/12840","heartbeat_back_addr":"172.21.15.78:6807/12840","heartbeat_front_addr":"172.21.15.78:6805/12840","state":["exists","up"]},{"osd":1,"uuid":"5d4eefd6-6f2f-4226-932d-2676c17b1796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":12841},{"type":"v1","addr":"172.21.15.78:6809","nonce":12841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":12841},{"type":"v1","addr":"172.21.15.78:6811","nonce":12841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":12841},{"type":"v1","addr":"172.21.15.78:6815","nonce":12841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":12841},{"type":"v1","addr":"172.21.15.78:6813","nonce":12841}]},"public_addr":"172.21.15.78:6809/12841","cluster_addr":"172.21.15.78:6811/12841","heartbeat_back_addr":"172.21.15.78:6815/12841","heartbeat_front_addr":"172.21.15.78:6813/12841","state":["exists","up"]},{"osd":2,"uuid":"538f2dff-c3d6-460a-942c-fb4e18295585","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":12842},{"type":"v1","addr":"172.21.15.78:6817","nonce":12842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":12842},{"type":"v1","addr":"172.21.15.78:6819","nonce":12842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":12842},{"type":"v1","addr":"172.21.15.78:6823","nonce":12842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":12842},{"type":"v1","addr":"172.21.15.78:6821","nonce":12842}]},"public_addr":"172.21.15.78:6817/12842","cluster_addr":"172.21.15.78:6819/12842","heartbeat_back_addr":"172.21.15.78:6823/12842","heartbeat_front_addr":"172.21.15.78:6821/12842","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:12.947673+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:12.965073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:12.967267+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-04-04T09:29:21.212 INFO:tasks.ceph.ceph_manager.cluster1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-04-04T09:29:15.544468+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-04-04T09:29:19.545788+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '13', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-04T09:29:21.213 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd pool get device_health_metrics pg_num 2021-04-04T09:29:21.514 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2021-04-04T09:29:21.529 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd pool get rbd pg_num 2021-04-04T09:29:21.833 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 8 2021-04-04T09:29:21.848 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2021-04-04T09:29:21.849 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2021-04-04T09:29:21.849 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2021-04-04T09:29:22.192 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:22.206 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":12657},{"type":"v1","addr":"172.21.15.78:6825","nonce":12657}]},"active_addr":"172.21.15.78:6825/12657","active_change":"2021-04-04T09:29:15.455361+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":472197832}]},{"addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":669716708}]},{"addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":1304677619}]}]}} 2021-04-04T09:29:22.208 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2021-04-04T09:29:22.208 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2021-04-04T09:29:22.209 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2021-04-04T09:29:22.509 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:22.509 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":14,"fsid":"7fae7b04-7e84-4b67-8401-2f7942dda448","created":"2021-04-04T09:29:10.649819+0000","modified":"2021-04-04T09:29:21.886536+0000","last_up_change":"2021-04-04T09:29:14.833919+0000","last_in_change":"2021-04-04T09:29:11.824921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-04-04T09:29:15.544468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-04-04T09:29:19.545788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e6aeea0f-5441-4eab-8180-13833d6d9516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":12840},{"type":"v1","addr":"172.21.15.78:6801","nonce":12840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":12840},{"type":"v1","addr":"172.21.15.78:6803","nonce":12840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":12840},{"type":"v1","addr":"172.21.15.78:6807","nonce":12840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":12840},{"type":"v1","addr":"172.21.15.78:6805","nonce":12840}]},"public_addr":"172.21.15.78:6801/12840","cluster_addr":"172.21.15.78:6803/12840","heartbeat_back_addr":"172.21.15.78:6807/12840","heartbeat_front_addr":"172.21.15.78:6805/12840","state":["exists","up"]},{"osd":1,"uuid":"5d4eefd6-6f2f-4226-932d-2676c17b1796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":12841},{"type":"v1","addr":"172.21.15.78:6809","nonce":12841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":12841},{"type":"v1","addr":"172.21.15.78:6811","nonce":12841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":12841},{"type":"v1","addr":"172.21.15.78:6815","nonce":12841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":12841},{"type":"v1","addr":"172.21.15.78:6813","nonce":12841}]},"public_addr":"172.21.15.78:6809/12841","cluster_addr":"172.21.15.78:6811/12841","heartbeat_back_addr":"172.21.15.78:6815/12841","heartbeat_front_addr":"172.21.15.78:6813/12841","state":["exists","up"]},{"osd":2,"uuid":"538f2dff-c3d6-460a-942c-fb4e18295585","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":12842},{"type":"v1","addr":"172.21.15.78:6817","nonce":12842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":12842},{"type":"v1","addr":"172.21.15.78:6819","nonce":12842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":12842},{"type":"v1","addr":"172.21.15.78:6823","nonce":12842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":12842},{"type":"v1","addr":"172.21.15.78:6821","nonce":12842}]},"public_addr":"172.21.15.78:6817/12842","cluster_addr":"172.21.15.78:6819/12842","heartbeat_back_addr":"172.21.15.78:6823/12842","heartbeat_front_addr":"172.21.15.78:6821/12842","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:12.947673+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:12.965073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:12.967267+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-04-04T09:29:22.523 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2021-04-04T09:29:22.524 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2021-04-04T09:29:22.821 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:22.822 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":14,"fsid":"7fae7b04-7e84-4b67-8401-2f7942dda448","created":"2021-04-04T09:29:10.649819+0000","modified":"2021-04-04T09:29:21.886536+0000","last_up_change":"2021-04-04T09:29:14.833919+0000","last_in_change":"2021-04-04T09:29:11.824921+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-04-04T09:29:15.544468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-04-04T09:29:19.545788+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e6aeea0f-5441-4eab-8180-13833d6d9516","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":12840},{"type":"v1","addr":"172.21.15.78:6801","nonce":12840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":12840},{"type":"v1","addr":"172.21.15.78:6803","nonce":12840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":12840},{"type":"v1","addr":"172.21.15.78:6807","nonce":12840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":12840},{"type":"v1","addr":"172.21.15.78:6805","nonce":12840}]},"public_addr":"172.21.15.78:6801/12840","cluster_addr":"172.21.15.78:6803/12840","heartbeat_back_addr":"172.21.15.78:6807/12840","heartbeat_front_addr":"172.21.15.78:6805/12840","state":["exists","up"]},{"osd":1,"uuid":"5d4eefd6-6f2f-4226-932d-2676c17b1796","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":12841},{"type":"v1","addr":"172.21.15.78:6809","nonce":12841}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":12841},{"type":"v1","addr":"172.21.15.78:6811","nonce":12841}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":12841},{"type":"v1","addr":"172.21.15.78:6815","nonce":12841}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":12841},{"type":"v1","addr":"172.21.15.78:6813","nonce":12841}]},"public_addr":"172.21.15.78:6809/12841","cluster_addr":"172.21.15.78:6811/12841","heartbeat_back_addr":"172.21.15.78:6815/12841","heartbeat_front_addr":"172.21.15.78:6813/12841","state":["exists","up"]},{"osd":2,"uuid":"538f2dff-c3d6-460a-942c-fb4e18295585","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":12842},{"type":"v1","addr":"172.21.15.78:6817","nonce":12842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":12842},{"type":"v1","addr":"172.21.15.78:6819","nonce":12842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":12842},{"type":"v1","addr":"172.21.15.78:6823","nonce":12842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":12842},{"type":"v1","addr":"172.21.15.78:6821","nonce":12842}]},"public_addr":"172.21.15.78:6817/12842","cluster_addr":"172.21.15.78:6819/12842","heartbeat_back_addr":"172.21.15.78:6823/12842","heartbeat_front_addr":"172.21.15.78:6821/12842","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:12.947673+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:12.965073+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:12.967267+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-04-04T09:29:22.835 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2021-04-04T09:29:23.004 INFO:teuthology.orchestra.run.smithi078.stdout:34359738371 2021-04-04T09:29:23.005 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2021-04-04T09:29:23.175 INFO:teuthology.orchestra.run.smithi078.stdout:34359738371 2021-04-04T09:29:23.175 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2021-04-04T09:29:23.345 INFO:teuthology.orchestra.run.smithi078.stdout:34359738371 2021-04-04T09:29:23.346 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2021-04-04T09:29:23.645 INFO:teuthology.orchestra.run.smithi078.stdout:34359738369 2021-04-04T09:29:23.666 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738369 for osd.0 2021-04-04T09:29:24.669 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2021-04-04T09:29:24.978 INFO:teuthology.orchestra.run.smithi078.stdout:34359738371 2021-04-04T09:29:24.993 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.0 2021-04-04T09:29:24.994 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2021-04-04T09:29:25.303 INFO:teuthology.orchestra.run.smithi078.stdout:34359738371 2021-04-04T09:29:25.318 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.1 2021-04-04T09:29:25.318 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2021-04-04T09:29:25.620 INFO:teuthology.orchestra.run.smithi078.stdout:34359738371 2021-04-04T09:29:25.635 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.2 2021-04-04T09:29:25.636 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2021-04-04T09:29:25.636 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2021-04-04T09:29:25.943 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:25.945 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2021-04-04T09:29:25.956 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2021-04-04T09:29:25.466456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281098140,"kb_used":707368,"kb_used_data":280390788,"kb_used_omap":189,"kb_used_meta":0,"kb_avail":280390772,"statfs":{"total":287844495360,"available":287120150528,"internally_reserved":16384,"allocated":287120166912,"data_stored":287120166912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":193594,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.573853"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.889914+0000","last_change":"2021-04-04T09:29:20.887866+0000","last_active":"2021-04-04T09:29:20.889914+0000","last_peered":"2021-04-04T09:29:20.889914+0000","last_clean":"2021-04-04T09:29:20.889914+0000","last_became_active":"2021-04-04T09:29:20.887602+0000","last_became_peered":"2021-04-04T09:29:20.887602+0000","last_unstale":"2021-04-04T09:29:20.889914+0000","last_undegraded":"2021-04-04T09:29:20.889914+0000","last_fullsized":"2021-04-04T09:29:20.889914+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_clean_scrub_stamp":"2021-04-04T09:29:19.864329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.889933+0000","last_change":"2021-04-04T09:29:20.887928+0000","last_active":"2021-04-04T09:29:20.889933+0000","last_peered":"2021-04-04T09:29:20.889933+0000","last_clean":"2021-04-04T09:29:20.889933+0000","last_became_active":"2021-04-04T09:29:20.887099+0000","last_became_peered":"2021-04-04T09:29:20.887099+0000","last_unstale":"2021-04-04T09:29:20.889933+0000","last_undegraded":"2021-04-04T09:29:20.889933+0000","last_fullsized":"2021-04-04T09:29:20.889933+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_clean_scrub_stamp":"2021-04-04T09:29:19.864329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.889580+0000","last_change":"2021-04-04T09:29:20.887488+0000","last_active":"2021-04-04T09:29:20.889580+0000","last_peered":"2021-04-04T09:29:20.889580+0000","last_clean":"2021-04-04T09:29:20.889580+0000","last_became_active":"2021-04-04T09:29:20.887230+0000","last_became_peered":"2021-04-04T09:29:20.887230+0000","last_unstale":"2021-04-04T09:29:20.889580+0000","last_undegraded":"2021-04-04T09:29:20.889580+0000","last_fullsized":"2021-04-04T09:29:20.889580+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_clean_scrub_stamp":"2021-04-04T09:29:19.864329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.891436+0000","last_change":"2021-04-04T09:29:20.890576+0000","last_active":"2021-04-04T09:29:20.891436+0000","last_peered":"2021-04-04T09:29:20.891436+0000","last_clean":"2021-04-04T09:29:20.891436+0000","last_became_active":"2021-04-04T09:29:20.890329+0000","last_became_peered":"2021-04-04T09:29:20.890329+0000","last_unstale":"2021-04-04T09:29:20.891436+0000","last_undegraded":"2021-04-04T09:29:20.891436+0000","last_fullsized":"2021-04-04T09:29:20.891436+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_clean_scrub_stamp":"2021-04-04T09:29:19.864329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'1","reported_seq":"13","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.884837+0000","last_change":"2021-04-04T09:29:16.964959+0000","last_active":"2021-04-04T09:29:20.884837+0000","last_peered":"2021-04-04T09:29:20.884837+0000","last_clean":"2021-04-04T09:29:20.884837+0000","last_became_active":"2021-04-04T09:29:16.964457+0000","last_became_peered":"2021-04-04T09:29:16.964457+0000","last_unstale":"2021-04-04T09:29:20.884837+0000","last_undegraded":"2021-04-04T09:29:20.884837+0000","last_fullsized":"2021-04-04T09:29:20.884837+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:15.838486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:15.838486+0000","last_clean_scrub_stamp":"2021-04-04T09:29:15.838486+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.889672+0000","last_change":"2021-04-04T09:29:20.885638+0000","last_active":"2021-04-04T09:29:20.889672+0000","last_peered":"2021-04-04T09:29:20.889672+0000","last_clean":"2021-04-04T09:29:20.889672+0000","last_became_active":"2021-04-04T09:29:20.885342+0000","last_became_peered":"2021-04-04T09:29:20.885342+0000","last_unstale":"2021-04-04T09:29:20.889672+0000","last_undegraded":"2021-04-04T09:29:20.889672+0000","last_fullsized":"2021-04-04T09:29:20.889672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_clean_scrub_stamp":"2021-04-04T09:29:19.864329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.891536+0000","last_change":"2021-04-04T09:29:20.890672+0000","last_active":"2021-04-04T09:29:20.891536+0000","last_peered":"2021-04-04T09:29:20.891536+0000","last_clean":"2021-04-04T09:29:20.891536+0000","last_became_active":"2021-04-04T09:29:20.890446+0000","last_became_peered":"2021-04-04T09:29:20.890446+0000","last_unstale":"2021-04-04T09:29:20.891536+0000","last_undegraded":"2021-04-04T09:29:20.891536+0000","last_fullsized":"2021-04-04T09:29:20.891536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_clean_scrub_stamp":"2021-04-04T09:29:19.864329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.887422+0000","last_change":"2021-04-04T09:29:20.886687+0000","last_active":"2021-04-04T09:29:20.887422+0000","last_peered":"2021-04-04T09:29:20.887422+0000","last_clean":"2021-04-04T09:29:20.887422+0000","last_became_active":"2021-04-04T09:29:20.886298+0000","last_became_peered":"2021-04-04T09:29:20.886298+0000","last_unstale":"2021-04-04T09:29:20.887422+0000","last_undegraded":"2021-04-04T09:29:20.887422+0000","last_fullsized":"2021-04-04T09:29:20.887422+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_clean_scrub_stamp":"2021-04-04T09:29:19.864329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.890971+0000","last_change":"2021-04-04T09:29:20.890856+0000","last_active":"2021-04-04T09:29:20.890971+0000","last_peered":"2021-04-04T09:29:20.890971+0000","last_clean":"2021-04-04T09:29:20.890971+0000","last_became_active":"2021-04-04T09:29:20.890591+0000","last_became_peered":"2021-04-04T09:29:20.890591+0000","last_unstale":"2021-04-04T09:29:20.890971+0000","last_undegraded":"2021-04-04T09:29:20.890971+0000","last_fullsized":"2021-04-04T09:29:20.890971+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_clean_scrub_stamp":"2021-04-04T09:29:19.864329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235804,"kb_used_data":93463584,"kb_used_omap":43,"kb_used_meta":0,"kb_avail":93463576,"statfs":{"total":95948165120,"available":95706701824,"internally_reserved":8192,"allocated":95706710016,"data_stored":95706710016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44930,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235756,"kb_used_data":93463624,"kb_used_omap":65,"kb_used_meta":0,"kb_avail":93463624,"statfs":{"total":95948165120,"available":95706750976,"internally_reserved":0,"allocated":95706750976,"data_stored":95706750976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":66609,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235808,"kb_used_data":93463580,"kb_used_omap":80,"kb_used_meta":0,"kb_avail":93463572,"statfs":{"total":95948165120,"available":95706697728,"internally_reserved":8192,"allocated":95706705920,"data_stored":95706705920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":82055,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]}]}],"pool_statfs":[]}} 2021-04-04T09:29:25.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2021-04-04T09:29:26.260 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:26.262 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2021-04-04T09:29:26.273 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2021-04-04T09:29:25.466456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281098140,"kb_used":707368,"kb_used_data":280390788,"kb_used_omap":189,"kb_used_meta":0,"kb_avail":280390772,"statfs":{"total":287844495360,"available":287120150528,"internally_reserved":16384,"allocated":287120166912,"data_stored":287120166912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":193594,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.573853"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.889914+0000","last_change":"2021-04-04T09:29:20.887866+0000","last_active":"2021-04-04T09:29:20.889914+0000","last_peered":"2021-04-04T09:29:20.889914+0000","last_clean":"2021-04-04T09:29:20.889914+0000","last_became_active":"2021-04-04T09:29:20.887602+0000","last_became_peered":"2021-04-04T09:29:20.887602+0000","last_unstale":"2021-04-04T09:29:20.889914+0000","last_undegraded":"2021-04-04T09:29:20.889914+0000","last_fullsized":"2021-04-04T09:29:20.889914+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_clean_scrub_stamp":"2021-04-04T09:29:19.864329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.889933+0000","last_change":"2021-04-04T09:29:20.887928+0000","last_active":"2021-04-04T09:29:20.889933+0000","last_peered":"2021-04-04T09:29:20.889933+0000","last_clean":"2021-04-04T09:29:20.889933+0000","last_became_active":"2021-04-04T09:29:20.887099+0000","last_became_peered":"2021-04-04T09:29:20.887099+0000","last_unstale":"2021-04-04T09:29:20.889933+0000","last_undegraded":"2021-04-04T09:29:20.889933+0000","last_fullsized":"2021-04-04T09:29:20.889933+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_clean_scrub_stamp":"2021-04-04T09:29:19.864329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.889580+0000","last_change":"2021-04-04T09:29:20.887488+0000","last_active":"2021-04-04T09:29:20.889580+0000","last_peered":"2021-04-04T09:29:20.889580+0000","last_clean":"2021-04-04T09:29:20.889580+0000","last_became_active":"2021-04-04T09:29:20.887230+0000","last_became_peered":"2021-04-04T09:29:20.887230+0000","last_unstale":"2021-04-04T09:29:20.889580+0000","last_undegraded":"2021-04-04T09:29:20.889580+0000","last_fullsized":"2021-04-04T09:29:20.889580+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_clean_scrub_stamp":"2021-04-04T09:29:19.864329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.891436+0000","last_change":"2021-04-04T09:29:20.890576+0000","last_active":"2021-04-04T09:29:20.891436+0000","last_peered":"2021-04-04T09:29:20.891436+0000","last_clean":"2021-04-04T09:29:20.891436+0000","last_became_active":"2021-04-04T09:29:20.890329+0000","last_became_peered":"2021-04-04T09:29:20.890329+0000","last_unstale":"2021-04-04T09:29:20.891436+0000","last_undegraded":"2021-04-04T09:29:20.891436+0000","last_fullsized":"2021-04-04T09:29:20.891436+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_clean_scrub_stamp":"2021-04-04T09:29:19.864329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'1","reported_seq":"13","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.884837+0000","last_change":"2021-04-04T09:29:16.964959+0000","last_active":"2021-04-04T09:29:20.884837+0000","last_peered":"2021-04-04T09:29:20.884837+0000","last_clean":"2021-04-04T09:29:20.884837+0000","last_became_active":"2021-04-04T09:29:16.964457+0000","last_became_peered":"2021-04-04T09:29:16.964457+0000","last_unstale":"2021-04-04T09:29:20.884837+0000","last_undegraded":"2021-04-04T09:29:20.884837+0000","last_fullsized":"2021-04-04T09:29:20.884837+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:15.838486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:15.838486+0000","last_clean_scrub_stamp":"2021-04-04T09:29:15.838486+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.889672+0000","last_change":"2021-04-04T09:29:20.885638+0000","last_active":"2021-04-04T09:29:20.889672+0000","last_peered":"2021-04-04T09:29:20.889672+0000","last_clean":"2021-04-04T09:29:20.889672+0000","last_became_active":"2021-04-04T09:29:20.885342+0000","last_became_peered":"2021-04-04T09:29:20.885342+0000","last_unstale":"2021-04-04T09:29:20.889672+0000","last_undegraded":"2021-04-04T09:29:20.889672+0000","last_fullsized":"2021-04-04T09:29:20.889672+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_clean_scrub_stamp":"2021-04-04T09:29:19.864329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.891536+0000","last_change":"2021-04-04T09:29:20.890672+0000","last_active":"2021-04-04T09:29:20.891536+0000","last_peered":"2021-04-04T09:29:20.891536+0000","last_clean":"2021-04-04T09:29:20.891536+0000","last_became_active":"2021-04-04T09:29:20.890446+0000","last_became_peered":"2021-04-04T09:29:20.890446+0000","last_unstale":"2021-04-04T09:29:20.891536+0000","last_undegraded":"2021-04-04T09:29:20.891536+0000","last_fullsized":"2021-04-04T09:29:20.891536+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_clean_scrub_stamp":"2021-04-04T09:29:19.864329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.887422+0000","last_change":"2021-04-04T09:29:20.886687+0000","last_active":"2021-04-04T09:29:20.887422+0000","last_peered":"2021-04-04T09:29:20.887422+0000","last_clean":"2021-04-04T09:29:20.887422+0000","last_became_active":"2021-04-04T09:29:20.886298+0000","last_became_peered":"2021-04-04T09:29:20.886298+0000","last_unstale":"2021-04-04T09:29:20.887422+0000","last_undegraded":"2021-04-04T09:29:20.887422+0000","last_fullsized":"2021-04-04T09:29:20.887422+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_clean_scrub_stamp":"2021-04-04T09:29:19.864329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:20.890971+0000","last_change":"2021-04-04T09:29:20.890856+0000","last_active":"2021-04-04T09:29:20.890971+0000","last_peered":"2021-04-04T09:29:20.890971+0000","last_clean":"2021-04-04T09:29:20.890971+0000","last_became_active":"2021-04-04T09:29:20.890591+0000","last_became_peered":"2021-04-04T09:29:20.890591+0000","last_unstale":"2021-04-04T09:29:20.890971+0000","last_undegraded":"2021-04-04T09:29:20.890971+0000","last_fullsized":"2021-04-04T09:29:20.890971+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:19.864329+0000","last_clean_scrub_stamp":"2021-04-04T09:29:19.864329+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235804,"kb_used_data":93463584,"kb_used_omap":43,"kb_used_meta":0,"kb_avail":93463576,"statfs":{"total":95948165120,"available":95706701824,"internally_reserved":8192,"allocated":95706710016,"data_stored":95706710016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44930,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235756,"kb_used_data":93463624,"kb_used_omap":65,"kb_used_meta":0,"kb_avail":93463624,"statfs":{"total":95948165120,"available":95706750976,"internally_reserved":0,"allocated":95706750976,"data_stored":95706750976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":66609,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235808,"kb_used_data":93463580,"kb_used_omap":80,"kb_used_meta":0,"kb_avail":93463572,"statfs":{"total":95948165120,"available":95706697728,"internally_reserved":8192,"allocated":95706705920,"data_stored":95706705920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":82055,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.251}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]}]}],"pool_statfs":[]}} 2021-04-04T09:29:26.275 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2021-04-04T09:29:26.275 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2021-04-04T09:29:26.275 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2021-04-04T09:29:26.276 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2021-04-04T09:29:26.625 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:26.641 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-04-04T09:29:26.641 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2021-04-04T09:29:26.642 INFO:teuthology.run_tasks:Running task ceph... 2021-04-04T09:29:26.656 INFO:tasks.ceph:Creating ceph cluster cluster2... 2021-04-04T09:29:26.657 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-04-04T09:29:26.657 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2021-04-04_06:07:03-rbd-pacific-distro-basic-smithi/6019607', 'branch': 'pacific', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-workunit-policy-simple}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '6019607', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run the rbd_mirror_journal.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2021-04-04_06:07:03-rbd-pacific-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '18.04', 'overrides': {'admin_socket': {'branch': 'pacific'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b'}}, 'workunit': {'branch': 'pacific', 'sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 80, 'repo': 'https://github.com/ceph/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b', 'sleep_before_teardown': 0, 'suite': 'rbd', 'suite_branch': 'pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_f4fb41419a42bee43a2069637ef17064073bf30b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b', 'targets': {'smithi078.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCox+hBSbqhZEOOs8jJK9PcXf99OuJixxSDbvBBJ8g0Jqe3aOnAMO+buNziqMRwbKJXzwgNnWt9t7jtlfJE+H+CfWSV1mTaaI4R3SGDIdVGXHuu38Y7B1XVImZeAvGYrlvVuquCp6oLBt4bucRosTI7NuXdLkm6f6DCOQJkhV09jum2gvIT0VtQrSEAFKIDU129Cr1q9zKd+7mHZwHM2KrT0974/I2dvr5qXy+6MKfDRd5P8wwKoVtlinNbVpPZ6NVyIKm5aOa+QOE7W2jIV/ObgS34+grwZ3kDeE5GvqcI7tZErKB5M5zat6zk125M+fyQ/xSBbE6I70tQ3kqoSDC7', 'smithi122.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0B/RVSuPh8DWhZMw8FntzsniAdnj1e24NvMQ7NpKZhgVshkWgPW0KytLhlnNC+PusZl6HqGhurZeByzBqf8oZwNNJa03FA9WRMHEURT6LJnP54V4HePaqUTn8mv8dALIvAyKSBiRLhEc99GOpKnJhwOqnLi/QimZerlOBSQbyNgtCdsZnKCpmlHpXI6NWwK1VzlblAhCLMnGdlLzCDQW5NJ53DfbIZNCFUX3r7Xn2bHzJyYCKVkIVIMhK4h0f0X+HxlkJ0zHzPrutH8Yzl96PUPn7TyttNXI3jhPpaaFXBGxk9OvHSxBvb87eIeWbaG8fz/xuz6DxcP6MazsCcwcn'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'f4fb41419a42bee43a2069637ef17064073bf30b'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_journal.sh']}, 'env': {'CEPH_ARGS': '', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1'}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '85d61eae4759f46ce21e9a37cd816a7a1a66c9d5', 'timestamp': '2021-04-04_06:07:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.13277'} 2021-04-04T09:29:26.658 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2021-04-04T09:29:26.661 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2021-04-04T09:29:26.670 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/run/ceph 2021-04-04T09:29:26.711 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/run/ceph 2021-04-04T09:29:26.728 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:26.732 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2021-04-04T09:29:26.776 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-04-04T09:29:26.777 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-04-04T09:29:26.823 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-04-04T09:29:26.824 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-04-04T09:29:26.824 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 341 Links: 1 2021-04-04T09:29:26.825 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-04-04T09:29:26.825 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2021-04-04 09:26:25.823132305 +0000 2021-04-04T09:29:26.826 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2021-04-04 09:26:25.819134305 +0000 2021-04-04T09:29:26.826 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2021-04-04 09:26:25.819134305 +0000 2021-04-04T09:29:26.827 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2021-04-04T09:29:26.833 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2021-04-04T09:29:26.833 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2021-04-04T09:29:26.834 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.00036448 s, 1.4 MB/s 2021-04-04T09:29:26.838 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-04-04T09:29:26.887 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-04-04T09:29:26.888 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-04-04T09:29:26.888 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 347 Links: 1 2021-04-04T09:29:26.889 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-04-04T09:29:26.889 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2021-04-04 09:26:26.166960301 +0000 2021-04-04T09:29:26.889 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2021-04-04 09:26:26.047020303 +0000 2021-04-04T09:29:26.890 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2021-04-04 09:26:26.047020303 +0000 2021-04-04T09:29:26.890 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2021-04-04T09:29:26.898 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2021-04-04T09:29:26.899 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2021-04-04T09:29:26.899 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000357254 s, 1.4 MB/s 2021-04-04T09:29:26.902 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-04-04T09:29:26.951 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-04-04T09:29:26.952 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-04-04T09:29:26.952 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 354 Links: 1 2021-04-04T09:29:26.953 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-04-04T09:29:26.953 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2021-04-04 09:26:26.290898299 +0000 2021-04-04T09:29:26.954 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2021-04-04 09:26:26.286900300 +0000 2021-04-04T09:29:26.954 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2021-04-04 09:26:26.286900300 +0000 2021-04-04T09:29:26.955 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2021-04-04T09:29:26.962 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2021-04-04T09:29:26.963 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2021-04-04T09:29:26.963 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000369932 s, 1.4 MB/s 2021-04-04T09:29:26.966 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-04-04T09:29:27.015 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-04-04T09:29:27.015 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-04-04T09:29:27.015 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 360 Links: 1 2021-04-04T09:29:27.016 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-04-04T09:29:27.016 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2021-04-04 09:26:26.630728296 +0000 2021-04-04T09:29:27.016 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2021-04-04 09:26:26.522782297 +0000 2021-04-04T09:29:27.016 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2021-04-04 09:26:26.522782297 +0000 2021-04-04T09:29:27.017 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2021-04-04T09:29:27.024 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2021-04-04T09:29:27.025 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2021-04-04T09:29:27.025 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000412948 s, 1.2 MB/s 2021-04-04T09:29:27.029 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2021-04-04T09:29:27.030 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-04-04T09:29:27.030 INFO:tasks.ceph:Generating config... 2021-04-04T09:29:27.031 INFO:tasks.ceph:[client] debug rbd = 20 2021-04-04T09:29:27.032 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2021-04-04T09:29:27.032 INFO:tasks.ceph:[client] log to stderr = False 2021-04-04T09:29:27.032 INFO:tasks.ceph:[client] rbd default features = 125 2021-04-04T09:29:27.032 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2021-04-04T09:29:27.033 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.033 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.033 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.034 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.034 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.034 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.035 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.035 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.035 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.035 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.036 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.036 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.036 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.037 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.037 INFO:tasks.ceph:[global] mon client directed command retry = 5 2021-04-04T09:29:27.037 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2021-04-04T09:29:27.038 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-04-04T09:29:27.038 INFO:tasks.ceph:[mgr] debug ms = 1 2021-04-04T09:29:27.038 INFO:tasks.ceph:[mon] debug mon = 20 2021-04-04T09:29:27.038 INFO:tasks.ceph:[mon] debug ms = 1 2021-04-04T09:29:27.039 INFO:tasks.ceph:[mon] debug paxos = 20 2021-04-04T09:29:27.039 INFO:tasks.ceph:[osd] debug ms = 1 2021-04-04T09:29:27.039 INFO:tasks.ceph:[osd] debug osd = 20 2021-04-04T09:29:27.040 INFO:tasks.ceph:[osd] osd objectstore = filestore 2021-04-04T09:29:27.040 INFO:tasks.ceph:[osd] osd sloppy crc = True 2021-04-04T09:29:27.040 INFO:tasks.ceph:Setting up cluster2.mon.a... 2021-04-04T09:29:27.041 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2021-04-04T09:29:27.104 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.keyring 2021-04-04T09:29:27.107 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2021-04-04T09:29:27.182 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2021-04-04T09:29:27.238 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.122')] 2021-04-04T09:29:27.239 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.122', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd sloppy crc': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2021-04-04T09:29:27.241 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:27.242 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-04-04T09:29:27.287 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.122 --print /home/ubuntu/cephtest/cluster2.monmap 2021-04-04T09:29:27.359 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2021-04-04T09:29:27.359 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: generated fsid 0fed4950-22e0-476b-9583-b4115f5b795c 2021-04-04T09:29:27.359 INFO:teuthology.orchestra.run.smithi122.stdout:epoch 0 2021-04-04T09:29:27.360 INFO:teuthology.orchestra.run.smithi122.stdout:fsid 0fed4950-22e0-476b-9583-b4115f5b795c 2021-04-04T09:29:27.360 INFO:teuthology.orchestra.run.smithi122.stdout:last_changed 2021-04-04T09:29:27.362356+0000 2021-04-04T09:29:27.360 INFO:teuthology.orchestra.run.smithi122.stdout:created 2021-04-04T09:29:27.362356+0000 2021-04-04T09:29:27.360 INFO:teuthology.orchestra.run.smithi122.stdout:min_mon_release 0 (unknown) 2021-04-04T09:29:27.361 INFO:teuthology.orchestra.run.smithi122.stdout:election_strategy: 1 2021-04-04T09:29:27.361 INFO:teuthology.orchestra.run.smithi122.stdout:0: [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon.a 2021-04-04T09:29:27.361 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2021-04-04T09:29:27.362 DEBUG:teuthology.orchestra.run.smithi122:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-04-04T09:29:27.411 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 0fed4950-22e0-476b-9583-b4115f5b795c... 2021-04-04T09:29:27.414 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2021-04-04T09:29:27.418 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2021-04-04T09:29:27.455 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2021-04-04T09:29:27.455 INFO:teuthology.orchestra.run.smithi078.stdout: chdir = "" 2021-04-04T09:29:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-04-04T09:29:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: auth supported = cephx 2021-04-04T09:29:27.456 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: filestore xattr use omap = true 2021-04-04T09:29:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: mon clock drift allowed = 1.000 2021-04-04T09:29:27.457 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: osd crush chooseleaf type = 0 2021-04-04T09:29:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: auth debug = true 2021-04-04T09:29:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on old message = true 2021-04-04T09:29:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on bug = true 2021-04-04T09:29:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: mon max pg per osd = 10000 # >= luminous 2021-04-04T09:29:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: mon pg warn max object skew = 0 2021-04-04T09:29:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.459 INFO:teuthology.orchestra.run.smithi078.stdout: # disable pg_autoscaler by default for new pools 2021-04-04T09:29:27.460 INFO:teuthology.orchestra.run.smithi078.stdout: osd_pool_default_pg_autoscale_mode = off 2021-04-04T09:29:27.460 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.460 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default size = 2 2021-04-04T09:29:27.460 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.461 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow primary affinity = true 2021-04-04T09:29:27.461 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow pg remap = true 2021-04-04T09:29:27.461 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on legacy crush tunables = false 2021-04-04T09:29:27.461 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on crush straw calc version zero = false 2021-04-04T09:29:27.461 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on no sortbitwise = false 2021-04-04T09:29:27.462 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on osd down out interval zero = false 2021-04-04T09:29:27.462 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on too few osds = false 2021-04-04T09:29:27.462 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-04-04T09:29:27.462 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_no_redundancy = false 2021-04-04T09:29:27.462 INFO:teuthology.orchestra.run.smithi078.stdout: mon_allow_pool_size_one = true 2021-04-04T09:29:27.463 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.463 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-04-04T09:29:27.463 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.464 INFO:teuthology.orchestra.run.smithi078.stdout: osd default data pool replay window = 5 2021-04-04T09:29:27.464 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.464 INFO:teuthology.orchestra.run.smithi078.stdout: mon allow pool delete = true 2021-04-04T09:29:27.464 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.465 INFO:teuthology.orchestra.run.smithi078.stdout: mon cluster log file level = debug 2021-04-04T09:29:27.465 INFO:teuthology.orchestra.run.smithi078.stdout: debug asserts on shutdown = true 2021-04-04T09:29:27.465 INFO:teuthology.orchestra.run.smithi078.stdout: mon health detail to clog = false 2021-04-04T09:29:27.466 INFO:teuthology.orchestra.run.smithi078.stdout: mon host = 172.21.15.122 2021-04-04T09:29:27.466 INFO:teuthology.orchestra.run.smithi078.stdout: mon client directed command retry = 5 2021-04-04T09:29:27.466 INFO:teuthology.orchestra.run.smithi078.stdout: ms inject socket failures = 5000 2021-04-04T09:29:27.467 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = 0fed4950-22e0-476b-9583-b4115f5b795c 2021-04-04T09:29:27.467 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.467 INFO:teuthology.orchestra.run.smithi078.stdout:[osd] 2021-04-04T09:29:27.467 INFO:teuthology.orchestra.run.smithi078.stdout: osd journal size = 100 2021-04-04T09:29:27.467 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.468 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub load threshold = 5.0 2021-04-04T09:29:27.468 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub max interval = 600 2021-04-04T09:29:27.468 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.468 INFO:teuthology.orchestra.run.smithi078.stdout: osd recover clone overlap = true 2021-04-04T09:29:27.469 INFO:teuthology.orchestra.run.smithi078.stdout: osd recovery max chunk = 1048576 2021-04-04T09:29:27.469 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.470 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug shutdown = true 2021-04-04T09:29:27.470 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug op order = true 2021-04-04T09:29:27.470 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify stray on activate = true 2021-04-04T09:29:27.470 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.470 INFO:teuthology.orchestra.run.smithi078.stdout: osd open classes on start = true 2021-04-04T09:29:27.471 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug pg log writeout = true 2021-04-04T09:29:27.471 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.472 INFO:teuthology.orchestra.run.smithi078.stdout: osd deep scrub update digest min age = 30 2021-04-04T09:29:27.472 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.472 INFO:teuthology.orchestra.run.smithi078.stdout: osd map max advance = 10 2021-04-04T09:29:27.472 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.472 INFO:teuthology.orchestra.run.smithi078.stdout: journal zero on create = true 2021-04-04T09:29:27.473 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.473 INFO:teuthology.orchestra.run.smithi078.stdout: filestore ondisk finisher threads = 3 2021-04-04T09:29:27.473 INFO:teuthology.orchestra.run.smithi078.stdout: filestore apply finisher threads = 3 2021-04-04T09:29:27.473 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.474 INFO:teuthology.orchestra.run.smithi078.stdout: bdev debug aio = true 2021-04-04T09:29:27.474 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug misdirected ops = true 2021-04-04T09:29:27.474 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2021-04-04T09:29:27.474 INFO:teuthology.orchestra.run.smithi078.stdout: debug osd = 20 2021-04-04T09:29:27.475 INFO:teuthology.orchestra.run.smithi078.stdout: osd objectstore = filestore 2021-04-04T09:29:27.475 INFO:teuthology.orchestra.run.smithi078.stdout: osd sloppy crc = True 2021-04-04T09:29:27.475 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.475 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr] 2021-04-04T09:29:27.476 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2021-04-04T09:29:27.476 INFO:teuthology.orchestra.run.smithi078.stdout: debug mgr = 20 2021-04-04T09:29:27.476 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2021-04-04T09:29:27.476 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2021-04-04T09:29:27.476 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2021-04-04T09:29:27.477 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2021-04-04T09:29:27.477 INFO:teuthology.orchestra.run.smithi078.stdout: mgr/telemetry/nag = false 2021-04-04T09:29:27.477 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.477 INFO:teuthology.orchestra.run.smithi078.stdout:[mon] 2021-04-04T09:29:27.478 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2021-04-04T09:29:27.478 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2021-04-04T09:29:27.478 INFO:teuthology.orchestra.run.smithi078.stdout: debug paxos = 20 2021-04-04T09:29:27.478 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2021-04-04T09:29:27.478 INFO:teuthology.orchestra.run.smithi078.stdout: mon data avail warn = 5 2021-04-04T09:29:27.479 INFO:teuthology.orchestra.run.smithi078.stdout: mon mgr mkfs grace = 240 2021-04-04T09:29:27.479 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2021-04-04T09:29:27.479 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd reporter subtree level = osd 2021-04-04T09:29:27.479 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd prime pg temp = true 2021-04-04T09:29:27.480 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2021-04-04T09:29:27.480 INFO:teuthology.orchestra.run.smithi078.stdout: 2021-04-04T09:29:27.480 INFO:teuthology.orchestra.run.smithi078.stdout:[client] 2021-04-04T09:29:27.480 INFO:teuthology.orchestra.run.smithi078.stdout: rgw cache enabled = true 2021-04-04T09:29:27.481 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable ops log = true 2021-04-04T09:29:27.481 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable usage log = true 2021-04-04T09:29:27.481 INFO:teuthology.orchestra.run.smithi078.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-04-04T09:29:27.481 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-04-04T09:29:27.481 INFO:teuthology.orchestra.run.smithi078.stdout: debug rbd = 20 2021-04-04T09:29:27.482 INFO:teuthology.orchestra.run.smithi078.stdout: debug rbd_mirror = 15 2021-04-04T09:29:27.482 INFO:teuthology.orchestra.run.smithi078.stdout: log to stderr = False 2021-04-04T09:29:27.482 INFO:teuthology.orchestra.run.smithi078.stdout: rbd default features = 125 2021-04-04T09:29:27.482 INFO:teuthology.orchestra.run.smithi078.stdout: rbd mirror image policy type = simple 2021-04-04T09:29:27.483 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.a] 2021-04-04T09:29:27.483 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.0] 2021-04-04T09:29:27.483 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.483 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.484 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.1] 2021-04-04T09:29:27.484 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.484 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.484 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.2] 2021-04-04T09:29:27.484 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.485 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.485 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.3] 2021-04-04T09:29:27.485 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.485 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.485 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.4] 2021-04-04T09:29:27.486 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.486 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.486 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.5] 2021-04-04T09:29:27.486 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.487 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.487 INFO:teuthology.orchestra.run.smithi078.stdout:[client.mirror.6] 2021-04-04T09:29:27.487 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.487 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.488 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2021-04-04T09:29:27.488 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2021-04-04T09:29:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-04-04T09:29:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2021-04-04T09:29:27.489 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.490 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2021-04-04T09:29:27.490 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.490 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2021-04-04T09:29:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2021-04-04T09:29:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2021-04-04T09:29:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2021-04-04T09:29:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2021-04-04T09:29:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.492 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2021-04-04T09:29:27.493 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2021-04-04T09:29:27.493 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.493 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2021-04-04T09:29:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2021-04-04T09:29:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2021-04-04T09:29:27.494 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2021-04-04T09:29:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2021-04-04T09:29:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2021-04-04T09:29:27.495 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2021-04-04T09:29:27.496 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2021-04-04T09:29:27.496 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2021-04-04T09:29:27.496 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2021-04-04T09:29:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-04-04T09:29:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2021-04-04T09:29:27.497 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2021-04-04T09:29:27.498 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.498 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-04-04T09:29:27.498 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2021-04-04T09:29:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.499 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2021-04-04T09:29:27.500 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.501 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2021-04-04T09:29:27.501 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2021-04-04T09:29:27.501 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2021-04-04T09:29:27.502 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = 172.21.15.122 2021-04-04T09:29:27.502 INFO:teuthology.orchestra.run.smithi122.stdout: mon client directed command retry = 5 2021-04-04T09:29:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: ms inject socket failures = 5000 2021-04-04T09:29:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = 0fed4950-22e0-476b-9583-b4115f5b795c 2021-04-04T09:29:27.503 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.504 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2021-04-04T09:29:27.504 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2021-04-04T09:29:27.505 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.505 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2021-04-04T09:29:27.505 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2021-04-04T09:29:27.506 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.506 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2021-04-04T09:29:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2021-04-04T09:29:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.507 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2021-04-04T09:29:27.508 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2021-04-04T09:29:27.508 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2021-04-04T09:29:27.509 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.509 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2021-04-04T09:29:27.509 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2021-04-04T09:29:27.510 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.510 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2021-04-04T09:29:27.511 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.511 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2021-04-04T09:29:27.511 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.512 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2021-04-04T09:29:27.512 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.513 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2021-04-04T09:29:27.513 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2021-04-04T09:29:27.513 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.514 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2021-04-04T09:29:27.514 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2021-04-04T09:29:27.515 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2021-04-04T09:29:27.515 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2021-04-04T09:29:27.515 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = filestore 2021-04-04T09:29:27.516 INFO:teuthology.orchestra.run.smithi122.stdout: osd sloppy crc = True 2021-04-04T09:29:27.516 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.517 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2021-04-04T09:29:27.517 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2021-04-04T09:29:27.517 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2021-04-04T09:29:27.518 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2021-04-04T09:29:27.518 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2021-04-04T09:29:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2021-04-04T09:29:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2021-04-04T09:29:27.519 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2021-04-04T09:29:27.520 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.520 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2021-04-04T09:29:27.521 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2021-04-04T09:29:27.521 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2021-04-04T09:29:27.522 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2021-04-04T09:29:27.522 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2021-04-04T09:29:27.523 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2021-04-04T09:29:27.523 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2021-04-04T09:29:27.523 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2021-04-04T09:29:27.524 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2021-04-04T09:29:27.524 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2021-04-04T09:29:27.525 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2021-04-04T09:29:27.525 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:27.525 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2021-04-04T09:29:27.526 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2021-04-04T09:29:27.526 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2021-04-04T09:29:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2021-04-04T09:29:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-04-04T09:29:27.527 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-04-04T09:29:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: debug rbd = 20 2021-04-04T09:29:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: debug rbd_mirror = 15 2021-04-04T09:29:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: log to stderr = False 2021-04-04T09:29:27.528 INFO:teuthology.orchestra.run.smithi122.stdout: rbd default features = 125 2021-04-04T09:29:27.529 INFO:teuthology.orchestra.run.smithi122.stdout: rbd mirror image policy type = simple 2021-04-04T09:29:27.529 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2021-04-04T09:29:27.529 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.0] 2021-04-04T09:29:27.529 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.530 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.530 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.1] 2021-04-04T09:29:27.530 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.530 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.530 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.2] 2021-04-04T09:29:27.531 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.531 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.531 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.3] 2021-04-04T09:29:27.531 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.532 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.532 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.4] 2021-04-04T09:29:27.532 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.533 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.533 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.5] 2021-04-04T09:29:27.533 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.533 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.533 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.6] 2021-04-04T09:29:27.534 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2021-04-04T09:29:27.534 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2021-04-04T09:29:27.535 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2021-04-04T09:29:27.536 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2021-04-04T09:29:27.622 INFO:tasks.ceph:Copying monmap to all nodes... 2021-04-04T09:29:27.623 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:27.624 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2021-04-04T09:29:27.631 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:27.632 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2021-04-04T09:29:27.679 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi078.front.sepia.ceph.com 2021-04-04T09:29:27.679 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:27.680 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/cluster2.keyring 2021-04-04T09:29:27.680 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2021-04-04T09:29:27.705 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:27.706 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2021-04-04T09:29:27.755 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi122.front.sepia.ceph.com 2021-04-04T09:29:27.755 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:27.755 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/cluster2.keyring 2021-04-04T09:29:27.756 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2021-04-04T09:29:27.781 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:27.781 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2021-04-04T09:29:27.830 INFO:tasks.ceph:Setting up mon nodes... 2021-04-04T09:29:27.831 INFO:tasks.ceph:Setting up mgr nodes... 2021-04-04T09:29:27.831 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2021-04-04T09:29:27.872 INFO:teuthology.orchestra.run.smithi078.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2021-04-04T09:29:27.875 INFO:tasks.ceph:Setting up mds nodes... 2021-04-04T09:29:27.875 INFO:tasks.ceph_client:Setting up client nodes... 2021-04-04T09:29:27.876 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2021-04-04T09:29:27.949 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/cluster2.client.0.keyring 2021-04-04T09:29:27.961 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2021-04-04T09:29:27.996 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2021-04-04T09:29:28.008 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2021-04-04T09:29:28.085 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2021-04-04T09:29:28.098 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2021-04-04T09:29:28.172 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2021-04-04T09:29:28.183 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2021-04-04T09:29:28.257 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2021-04-04T09:29:28.267 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2021-04-04T09:29:28.343 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2021-04-04T09:29:28.355 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2021-04-04T09:29:28.429 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2021-04-04T09:29:28.440 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2021-04-04T09:29:28.513 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2021-04-04T09:29:28.525 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2021-04-04T09:29:28.600 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2021-04-04T09:29:28.613 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-04-04T09:29:28.613 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-04-04T09:29:28.613 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2021-04-04T09:29:28.672 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2021-04-04T09:29:28.673 INFO:tasks.ceph:role: cluster2.osd.0 2021-04-04T09:29:28.673 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com 2021-04-04T09:29:28.674 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-04-04T09:29:29.198 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-04-04T09:29:29.199 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2021-04-04T09:29:29.199 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-04-04T09:29:29.200 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-04-04T09:29:29.200 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2021-04-04T09:29:29.200 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-04-04T09:29:29.200 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-04-04T09:29:29.200 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-04-04T09:29:29.201 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-04-04T09:29:29.202 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2021-04-04T09:29:29.202 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2021-04-04T09:29:29.274 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2021-04-04T09:29:29.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-04T09:29:29.286 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: /sbin/restorecon: command not found 2021-04-04T09:29:29.287 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2021-04-04T09:29:29.344 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2021-04-04T09:29:29.344 INFO:tasks.ceph:role: cluster2.osd.1 2021-04-04T09:29:29.344 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com 2021-04-04T09:29:29.345 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-04-04T09:29:29.869 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-04-04T09:29:29.869 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2021-04-04T09:29:29.869 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-04-04T09:29:29.870 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-04-04T09:29:29.870 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2021-04-04T09:29:29.870 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-04-04T09:29:29.870 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-04-04T09:29:29.871 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-04-04T09:29:29.871 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-04-04T09:29:29.872 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2021-04-04T09:29:29.872 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2021-04-04T09:29:29.927 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2021-04-04T09:29:29.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-04T09:29:29.937 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: /sbin/restorecon: command not found 2021-04-04T09:29:29.938 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2021-04-04T09:29:29.995 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2021-04-04T09:29:29.995 INFO:tasks.ceph:role: cluster2.osd.2 2021-04-04T09:29:29.996 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com 2021-04-04T09:29:29.996 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-04-04T09:29:30.486 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-04-04T09:29:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2021-04-04T09:29:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-04-04T09:29:30.487 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-04-04T09:29:30.487 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2021-04-04T09:29:30.488 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-04-04T09:29:30.488 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-04-04T09:29:30.488 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-04-04T09:29:30.488 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-04-04T09:29:30.489 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2021-04-04T09:29:30.490 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2021-04-04T09:29:30.538 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2021-04-04T09:29:30.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-04T09:29:30.550 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: /sbin/restorecon: command not found 2021-04-04T09:29:30.550 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2021-04-04T09:29:30.652 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:30.651+0000 7fa963eb2f00 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2021-04-04T09:29:30.652 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:30.651+0000 7fa963eb2f00 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2021-04-04T09:29:30.676 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:30.675+0000 7fa963eb2f00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:30.677 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:30.679+0000 7fa963eb2f00 -1 journal FileJournal::_open_file : zeroing journal 2021-04-04T09:29:30.901 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:30.903+0000 7fa963eb2f00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:30.905 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:30.903+0000 7fa963eb2f00 -1 journal do_read_entry(4096): bad header magic 2021-04-04T09:29:30.905 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:30.903+0000 7fa963eb2f00 -1 journal do_read_entry(4096): bad header magic 2021-04-04T09:29:30.921 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2021-04-04T09:29:30.934 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2021-04-04T09:29:31.013 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:31.015+0000 7fe61e977f00 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2021-04-04T09:29:31.014 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:31.015+0000 7fe61e977f00 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2021-04-04T09:29:31.054 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:31.055+0000 7fe61e977f00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:31.058 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:31.059+0000 7fe61e977f00 -1 journal FileJournal::_open_file : zeroing journal 2021-04-04T09:29:31.305 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:31.307+0000 7fe61e977f00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:31.309 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:31.311+0000 7fe61e977f00 -1 journal do_read_entry(4096): bad header magic 2021-04-04T09:29:31.310 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:31.311+0000 7fe61e977f00 -1 journal do_read_entry(4096): bad header magic 2021-04-04T09:29:31.335 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2021-04-04T09:29:31.392 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2021-04-04T09:29:31.474 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:31.475+0000 7f8e33973f00 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2021-04-04T09:29:31.477 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:31.479+0000 7f8e33973f00 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2021-04-04T09:29:31.511 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:31.511+0000 7f8e33973f00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:31.511 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:31.511+0000 7f8e33973f00 -1 journal FileJournal::_open_file : zeroing journal 2021-04-04T09:29:31.746 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:31.747+0000 7f8e33973f00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:31.748 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:31.747+0000 7f8e33973f00 -1 journal do_read_entry(4096): bad header magic 2021-04-04T09:29:31.749 INFO:teuthology.orchestra.run.smithi122.stderr:2021-04-04T09:29:31.747+0000 7f8e33973f00 -1 journal do_read_entry(4096): bad header magic 2021-04-04T09:29:31.770 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2021-04-04T09:29:31.829 INFO:tasks.ceph:Reading keys from all nodes... 2021-04-04T09:29:31.830 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:31.830 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2021-04-04T09:29:31.849 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:31.850 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2021-04-04T09:29:31.887 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:31.888 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2021-04-04T09:29:31.942 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:31.942 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2021-04-04T09:29:31.999 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2021-04-04T09:29:32.000 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2021-04-04T09:29:32.008 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:32.010 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2021-04-04T09:29:32.048 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:32.049 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2021-04-04T09:29:32.094 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:32.095 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2021-04-04T09:29:32.142 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:32.143 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2021-04-04T09:29:32.190 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:32.191 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2021-04-04T09:29:32.238 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:32.239 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2021-04-04T09:29:32.290 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:32.291 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2021-04-04T09:29:32.339 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:32.340 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2021-04-04T09:29:32.387 INFO:tasks.ceph:Adding keys to all mons... 2021-04-04T09:29:32.388 DEBUG:teuthology.orchestra.run.smithi122:> sudo tee -a /etc/ceph/cluster2.keyring 2021-04-04T09:29:32.441 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr.x] 2021-04-04T09:29:32.442 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQD3hmlgbNoHNBAAluToGOfKQ6GO6J1J00GbkQ== 2021-04-04T09:29:32.442 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.0] 2021-04-04T09:29:32.443 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQD6hmlgIGkOJxAAR64jGkLJrifQUQ5DOGrmjA== 2021-04-04T09:29:32.443 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.1] 2021-04-04T09:29:32.443 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQD7hmlgv80KARAAagrMVgBuJ+8QwbIQ24rm9Q== 2021-04-04T09:29:32.443 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.2] 2021-04-04T09:29:32.444 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQD7hmlgOBB9HBAAlNvUpmdHbyAg76oOssQoUw== 2021-04-04T09:29:32.444 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2021-04-04T09:29:32.444 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQD3hmlg9/eTOBAAU6qxWpLww3GPOOIWr4jbRg== 2021-04-04T09:29:32.444 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror] 2021-04-04T09:29:32.444 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQD3hmlgX+OUOxAAJE722EZMZGZ8ATPajom+6g== 2021-04-04T09:29:32.445 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.0] 2021-04-04T09:29:32.445 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQD4hmlgjlxHBRAAF2rRMYlWuLHmfRbDzReKtg== 2021-04-04T09:29:32.445 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.1] 2021-04-04T09:29:32.445 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQD4hmlgZJV3ChAA/6hdpVoGF5/pWeX4aA6FHg== 2021-04-04T09:29:32.446 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.2] 2021-04-04T09:29:32.446 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQD4hmlg4ECGDxAAH+5sdE+FNwWoczZq4KBfCg== 2021-04-04T09:29:32.446 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.3] 2021-04-04T09:29:32.446 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQD4hmlgxLqoFBAA+87ma+NBj3Swqy20fJR9tw== 2021-04-04T09:29:32.446 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.4] 2021-04-04T09:29:32.447 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQD4hmlgz2/DGRAA29IKME31oG7Es9GjgZt/hQ== 2021-04-04T09:29:32.447 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.5] 2021-04-04T09:29:32.447 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQD4hmlgkfnQHhAAApidYH04LdzljoZExLXnOg== 2021-04-04T09:29:32.447 INFO:teuthology.orchestra.run.smithi122.stdout:[client.mirror.6] 2021-04-04T09:29:32.447 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQD4hmlgYx8BJBAAQbeV0K/YsWyMd6n1mQIb6w== 2021-04-04T09:29:32.453 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-04-04T09:29:32.538 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-04T09:29:32.627 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-04T09:29:32.710 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-04-04T09:29:32.794 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:32.880 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:32.969 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:33.059 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:33.145 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:33.186 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:33.278 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:33.363 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:33.448 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-04-04T09:29:33.532 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-04-04T09:29:33.533 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2021-04-04T09:29:33.591 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2021-04-04T09:29:33.706 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2021-04-04T09:29:33.722 DEBUG:teuthology.orchestra.run.smithi122:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2021-04-04T09:29:33.770 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2021-04-04T09:29:33.770 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2021-04-04T09:29:33.771 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2021-04-04T09:29:33.814 INFO:tasks.ceph.cluster2.mon.a:Started 2021-04-04T09:29:33.814 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2021-04-04T09:29:33.815 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2021-04-04T09:29:33.815 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2021-04-04T09:29:33.819 INFO:tasks.ceph.cluster2.mgr.x:Started 2021-04-04T09:29:33.819 INFO:tasks.ceph:Setting crush tunables to default 2021-04-04T09:29:33.820 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster2 osd crush tunables default 2021-04-04T09:29:34.170 INFO:teuthology.orchestra.run.smithi122.stderr:adjusted tunables profile to default 2021-04-04T09:29:34.186 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2021-04-04T09:29:34.187 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:34.188 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2021-04-04T09:29:34.203 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:34.203 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2021-04-04T09:29:34.258 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:29:34.259 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2021-04-04T09:29:34.317 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster2 osd new 3ac716b8-c7e4-496b-afae-8e2d488ad1d4 0 2021-04-04T09:29:34.586 INFO:teuthology.orchestra.run.smithi122.stdout:0 2021-04-04T09:29:34.606 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster2 osd new 6fa78d70-6d62-447c-97d2-b715486a5b55 1 2021-04-04T09:29:34.830 INFO:teuthology.orchestra.run.smithi122.stdout:1 2021-04-04T09:29:34.844 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster2 osd new fc965760-5eed-4344-9228-7342e847a67c 2 2021-04-04T09:29:35.062 INFO:teuthology.orchestra.run.smithi122.stdout:2 2021-04-04T09:29:35.082 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2021-04-04T09:29:35.082 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2021-04-04T09:29:35.085 INFO:tasks.ceph.cluster2.osd.0:Started 2021-04-04T09:29:35.086 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2021-04-04T09:29:35.086 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2021-04-04T09:29:35.090 INFO:tasks.ceph.cluster2.osd.1:Started 2021-04-04T09:29:35.091 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2021-04-04T09:29:35.091 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2021-04-04T09:29:35.094 INFO:tasks.ceph.cluster2.osd.2:Started 2021-04-04T09:29:35.094 INFO:tasks.ceph:Waiting for OSDs to come up 2021-04-04T09:29:35.153 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2021-04-04T09:29:35.155+0000 7f6f5efdcf00 -1 Falling back to public interface 2021-04-04T09:29:35.157 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T09:29:35.159+0000 7f852061af00 -1 Falling back to public interface 2021-04-04T09:29:35.161 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T09:29:35.163+0000 7fc4d8515f00 -1 Falling back to public interface 2021-04-04T09:29:35.172 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2021-04-04T09:29:35.171+0000 7f6f5efdcf00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:35.173 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2021-04-04T09:29:35.171+0000 7f6f5efdcf00 -1 journal do_read_entry(8192): bad header magic 2021-04-04T09:29:35.173 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2021-04-04T09:29:35.171+0000 7f6f5efdcf00 -1 journal do_read_entry(8192): bad header magic 2021-04-04T09:29:35.173 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T09:29:35.175+0000 7f852061af00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:35.174 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T09:29:35.175+0000 7f852061af00 -1 journal do_read_entry(8192): bad header magic 2021-04-04T09:29:35.174 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T09:29:35.175+0000 7f852061af00 -1 journal do_read_entry(8192): bad header magic 2021-04-04T09:29:35.181 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T09:29:35.183+0000 7fc4d8515f00 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-04-04T09:29:35.181 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T09:29:35.183+0000 7fc4d8515f00 -1 journal do_read_entry(8192): bad header magic 2021-04-04T09:29:35.182 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T09:29:35.183+0000 7fc4d8515f00 -1 journal do_read_entry(8192): bad header magic 2021-04-04T09:29:35.191 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T09:29:35.191+0000 7f852061af00 -1 osd.0 0 log_to_monitors {default=true} 2021-04-04T09:29:35.191 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2021-04-04T09:29:35.191+0000 7f6f5efdcf00 -1 osd.1 0 log_to_monitors {default=true} 2021-04-04T09:29:35.200 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T09:29:35.199+0000 7fc4d8515f00 -1 osd.2 0 log_to_monitors {default=true} 2021-04-04T09:29:35.396 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2021-04-04T09:29:35.713 INFO:teuthology.misc.health.smithi122.stderr:2021-04-04T09:29:35.715+0000 7f397ffff700 0 --2- 172.21.15.122:0/3524774755 >> v2:172.21.15.122:3300/0 conn(0x7f39880a0d70 0x7f39880611f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2021-04-04T09:29:35.802 INFO:teuthology.misc.health.smithi122.stdout: 2021-04-04T09:29:35.803 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":5,"fsid":"0fed4950-22e0-476b-9583-b4115f5b795c","created":"2021-04-04T09:29:33.941318+0000","modified":"2021-04-04T09:29:35.062846+0000","last_up_change":"0.000000","last_in_change":"2021-04-04T09:29:35.062846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"3ac716b8-c7e4-496b-afae-8e2d488ad1d4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6fa78d70-6d62-447c-97d2-b715486a5b55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc965760-5eed-4344-9228-7342e847a67c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-04-04T09:29:35.815 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-04-04T09:29:37.069 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2021-04-04T09:29:37.071+0000 7f6f56ca5700 -1 osd.1 0 waiting for initial osdmap 2021-04-04T09:29:37.070 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T09:29:37.071+0000 7f85182e3700 -1 osd.0 0 waiting for initial osdmap 2021-04-04T09:29:37.071 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T09:29:37.071+0000 7fc4d01de700 -1 osd.2 0 waiting for initial osdmap 2021-04-04T09:29:42.119 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2021-04-04T09:29:42.418 INFO:teuthology.misc.health.smithi122.stdout: 2021-04-04T09:29:42.419 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":11,"fsid":"0fed4950-22e0-476b-9583-b4115f5b795c","created":"2021-04-04T09:29:33.941318+0000","modified":"2021-04-04T09:29:41.092897+0000","last_up_change":"2021-04-04T09:29:38.071514+0000","last_in_change":"2021-04-04T09:29:35.062846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-04-04T09:29:38.142144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"3ac716b8-c7e4-496b-afae-8e2d488ad1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":12798},{"type":"v1","addr":"172.21.15.122:6809","nonce":12798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":12798},{"type":"v1","addr":"172.21.15.122:6811","nonce":12798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":12798},{"type":"v1","addr":"172.21.15.122:6815","nonce":12798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":12798},{"type":"v1","addr":"172.21.15.122:6813","nonce":12798}]},"public_addr":"172.21.15.122:6809/12798","cluster_addr":"172.21.15.122:6811/12798","heartbeat_back_addr":"172.21.15.122:6815/12798","heartbeat_front_addr":"172.21.15.122:6813/12798","state":["exists","up"]},{"osd":1,"uuid":"6fa78d70-6d62-447c-97d2-b715486a5b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":12799},{"type":"v1","addr":"172.21.15.122:6801","nonce":12799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":12799},{"type":"v1","addr":"172.21.15.122:6803","nonce":12799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":12799},{"type":"v1","addr":"172.21.15.122:6807","nonce":12799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":12799},{"type":"v1","addr":"172.21.15.122:6805","nonce":12799}]},"public_addr":"172.21.15.122:6801/12799","cluster_addr":"172.21.15.122:6803/12799","heartbeat_back_addr":"172.21.15.122:6807/12799","heartbeat_front_addr":"172.21.15.122:6805/12799","state":["exists","up"]},{"osd":2,"uuid":"fc965760-5eed-4344-9228-7342e847a67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":12800},{"type":"v1","addr":"172.21.15.122:6817","nonce":12800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":12800},{"type":"v1","addr":"172.21.15.122:6819","nonce":12800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":12800},{"type":"v1","addr":"172.21.15.122:6823","nonce":12800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":12800},{"type":"v1","addr":"172.21.15.122:6821","nonce":12800}]},"public_addr":"172.21.15.122:6817/12800","cluster_addr":"172.21.15.122:6819/12800","heartbeat_back_addr":"172.21.15.122:6823/12800","heartbeat_front_addr":"172.21.15.122:6821/12800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:36.174179+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:36.215401+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:36.241235+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-04-04T09:29:42.433 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-04-04T09:29:42.433 INFO:tasks.ceph:Creating RBD pool 2021-04-04T09:29:42.434 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster2 osd pool create rbd 8 2021-04-04T09:29:43.128 INFO:teuthology.orchestra.run.smithi122.stderr:pool 'rbd' created 2021-04-04T09:29:43.142 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-04-04T09:29:44.126 INFO:teuthology.orchestra.run.smithi122.stderr:enabled application 'rbd' on pool 'rbd' 2021-04-04T09:29:44.139 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2021-04-04T09:29:44.139 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2021-04-04T09:29:44.140 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-04-04T09:29:44.430 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:44.430 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":13,"fsid":"0fed4950-22e0-476b-9583-b4115f5b795c","created":"2021-04-04T09:29:33.941318+0000","modified":"2021-04-04T09:29:44.117652+0000","last_up_change":"2021-04-04T09:29:38.071514+0000","last_in_change":"2021-04-04T09:29:35.062846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-04-04T09:29:38.142144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-04-04T09:29:42.801690+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"3ac716b8-c7e4-496b-afae-8e2d488ad1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":12798},{"type":"v1","addr":"172.21.15.122:6809","nonce":12798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":12798},{"type":"v1","addr":"172.21.15.122:6811","nonce":12798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":12798},{"type":"v1","addr":"172.21.15.122:6815","nonce":12798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":12798},{"type":"v1","addr":"172.21.15.122:6813","nonce":12798}]},"public_addr":"172.21.15.122:6809/12798","cluster_addr":"172.21.15.122:6811/12798","heartbeat_back_addr":"172.21.15.122:6815/12798","heartbeat_front_addr":"172.21.15.122:6813/12798","state":["exists","up"]},{"osd":1,"uuid":"6fa78d70-6d62-447c-97d2-b715486a5b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":12799},{"type":"v1","addr":"172.21.15.122:6801","nonce":12799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":12799},{"type":"v1","addr":"172.21.15.122:6803","nonce":12799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":12799},{"type":"v1","addr":"172.21.15.122:6807","nonce":12799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":12799},{"type":"v1","addr":"172.21.15.122:6805","nonce":12799}]},"public_addr":"172.21.15.122:6801/12799","cluster_addr":"172.21.15.122:6803/12799","heartbeat_back_addr":"172.21.15.122:6807/12799","heartbeat_front_addr":"172.21.15.122:6805/12799","state":["exists","up"]},{"osd":2,"uuid":"fc965760-5eed-4344-9228-7342e847a67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":12800},{"type":"v1","addr":"172.21.15.122:6817","nonce":12800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":12800},{"type":"v1","addr":"172.21.15.122:6819","nonce":12800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":12800},{"type":"v1","addr":"172.21.15.122:6823","nonce":12800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":12800},{"type":"v1","addr":"172.21.15.122:6821","nonce":12800}]},"public_addr":"172.21.15.122:6817/12800","cluster_addr":"172.21.15.122:6819/12800","heartbeat_back_addr":"172.21.15.122:6823/12800","heartbeat_front_addr":"172.21.15.122:6821/12800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:36.174179+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:36.215401+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:36.241235+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-04-04T09:29:44.444 INFO:tasks.ceph.ceph_manager.cluster2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-04-04T09:29:38.142144+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-04-04T09:29:42.801690+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '13', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-04-04T09:29:44.445 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd pool get device_health_metrics pg_num 2021-04-04T09:29:44.749 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 1 2021-04-04T09:29:44.763 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd pool get rbd pg_num 2021-04-04T09:29:45.076 INFO:teuthology.orchestra.run.smithi122.stdout:pg_num: 8 2021-04-04T09:29:45.095 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2021-04-04T09:29:45.096 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2021-04-04T09:29:45.096 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2021-04-04T09:29:45.436 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:45.454 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":13834},{"type":"v1","addr":"172.21.15.78:6827","nonce":13834}]},"active_addr":"172.21.15.78:6827/13834","active_change":"2021-04-04T09:29:37.986551+0000","active_mgr_features":4540138297136906239,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":1544566484}]},{"addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":275653177}]}]}} 2021-04-04T09:29:45.456 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2021-04-04T09:29:45.457 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2021-04-04T09:29:45.457 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2021-04-04T09:29:45.754 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:45.755 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":14,"fsid":"0fed4950-22e0-476b-9583-b4115f5b795c","created":"2021-04-04T09:29:33.941318+0000","modified":"2021-04-04T09:29:45.123581+0000","last_up_change":"2021-04-04T09:29:38.071514+0000","last_in_change":"2021-04-04T09:29:35.062846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-04-04T09:29:38.142144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-04-04T09:29:42.801690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"3ac716b8-c7e4-496b-afae-8e2d488ad1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":12798},{"type":"v1","addr":"172.21.15.122:6809","nonce":12798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":12798},{"type":"v1","addr":"172.21.15.122:6811","nonce":12798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":12798},{"type":"v1","addr":"172.21.15.122:6815","nonce":12798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":12798},{"type":"v1","addr":"172.21.15.122:6813","nonce":12798}]},"public_addr":"172.21.15.122:6809/12798","cluster_addr":"172.21.15.122:6811/12798","heartbeat_back_addr":"172.21.15.122:6815/12798","heartbeat_front_addr":"172.21.15.122:6813/12798","state":["exists","up"]},{"osd":1,"uuid":"6fa78d70-6d62-447c-97d2-b715486a5b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":12799},{"type":"v1","addr":"172.21.15.122:6801","nonce":12799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":12799},{"type":"v1","addr":"172.21.15.122:6803","nonce":12799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":12799},{"type":"v1","addr":"172.21.15.122:6807","nonce":12799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":12799},{"type":"v1","addr":"172.21.15.122:6805","nonce":12799}]},"public_addr":"172.21.15.122:6801/12799","cluster_addr":"172.21.15.122:6803/12799","heartbeat_back_addr":"172.21.15.122:6807/12799","heartbeat_front_addr":"172.21.15.122:6805/12799","state":["exists","up"]},{"osd":2,"uuid":"fc965760-5eed-4344-9228-7342e847a67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":12800},{"type":"v1","addr":"172.21.15.122:6817","nonce":12800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":12800},{"type":"v1","addr":"172.21.15.122:6819","nonce":12800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":12800},{"type":"v1","addr":"172.21.15.122:6823","nonce":12800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":12800},{"type":"v1","addr":"172.21.15.122:6821","nonce":12800}]},"public_addr":"172.21.15.122:6817/12800","cluster_addr":"172.21.15.122:6819/12800","heartbeat_back_addr":"172.21.15.122:6823/12800","heartbeat_front_addr":"172.21.15.122:6821/12800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:36.174179+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:36.215401+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:36.241235+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-04-04T09:29:45.769 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2021-04-04T09:29:45.770 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2021-04-04T09:29:46.063 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:46.064 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":14,"fsid":"0fed4950-22e0-476b-9583-b4115f5b795c","created":"2021-04-04T09:29:33.941318+0000","modified":"2021-04-04T09:29:45.123581+0000","last_up_change":"2021-04-04T09:29:38.071514+0000","last_in_change":"2021-04-04T09:29:35.062846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-04-04T09:29:38.142144+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-04-04T09:29:42.801690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"3ac716b8-c7e4-496b-afae-8e2d488ad1d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":12798},{"type":"v1","addr":"172.21.15.122:6809","nonce":12798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":12798},{"type":"v1","addr":"172.21.15.122:6811","nonce":12798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":12798},{"type":"v1","addr":"172.21.15.122:6815","nonce":12798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":12798},{"type":"v1","addr":"172.21.15.122:6813","nonce":12798}]},"public_addr":"172.21.15.122:6809/12798","cluster_addr":"172.21.15.122:6811/12798","heartbeat_back_addr":"172.21.15.122:6815/12798","heartbeat_front_addr":"172.21.15.122:6813/12798","state":["exists","up"]},{"osd":1,"uuid":"6fa78d70-6d62-447c-97d2-b715486a5b55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":12799},{"type":"v1","addr":"172.21.15.122:6801","nonce":12799}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":12799},{"type":"v1","addr":"172.21.15.122:6803","nonce":12799}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":12799},{"type":"v1","addr":"172.21.15.122:6807","nonce":12799}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":12799},{"type":"v1","addr":"172.21.15.122:6805","nonce":12799}]},"public_addr":"172.21.15.122:6801/12799","cluster_addr":"172.21.15.122:6803/12799","heartbeat_back_addr":"172.21.15.122:6807/12799","heartbeat_front_addr":"172.21.15.122:6805/12799","state":["exists","up"]},{"osd":2,"uuid":"fc965760-5eed-4344-9228-7342e847a67c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":12800},{"type":"v1","addr":"172.21.15.122:6817","nonce":12800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":12800},{"type":"v1","addr":"172.21.15.122:6819","nonce":12800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":12800},{"type":"v1","addr":"172.21.15.122:6823","nonce":12800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":12800},{"type":"v1","addr":"172.21.15.122:6821","nonce":12800}]},"public_addr":"172.21.15.122:6817/12800","cluster_addr":"172.21.15.122:6819/12800","heartbeat_back_addr":"172.21.15.122:6823/12800","heartbeat_front_addr":"172.21.15.122:6821/12800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:36.174179+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:36.215401+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138297136906239,"old_weight":0,"last_purged_snaps_scrub":"2021-04-04T09:29:36.241235+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-04-04T09:29:46.078 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2021-04-04T09:29:46.246 INFO:teuthology.orchestra.run.smithi122.stdout:34359738371 2021-04-04T09:29:46.247 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2021-04-04T09:29:46.415 INFO:teuthology.orchestra.run.smithi122.stdout:34359738371 2021-04-04T09:29:46.415 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2021-04-04T09:29:46.585 INFO:teuthology.orchestra.run.smithi122.stdout:34359738371 2021-04-04T09:29:46.586 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2021-04-04T09:29:46.889 INFO:teuthology.orchestra.run.smithi122.stdout:34359738370 2021-04-04T09:29:46.904 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738370 for osd.0 2021-04-04T09:29:47.906 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2021-04-04T09:29:48.220 INFO:teuthology.orchestra.run.smithi122.stdout:34359738371 2021-04-04T09:29:48.235 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738371 for osd.0 2021-04-04T09:29:48.236 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2021-04-04T09:29:48.530 INFO:teuthology.orchestra.run.smithi122.stdout:34359738371 2021-04-04T09:29:48.545 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738371 for osd.1 2021-04-04T09:29:48.545 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2021-04-04T09:29:48.846 INFO:teuthology.orchestra.run.smithi122.stdout:34359738371 2021-04-04T09:29:48.860 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738371 for osd.2 2021-04-04T09:29:48.861 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2021-04-04T09:29:48.861 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2021-04-04T09:29:49.161 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:49.163 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2021-04-04T09:29:49.175 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-04-04T09:29:47.994189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281098140,"kb_used":707376,"kb_used_data":280390780,"kb_used_omap":189,"kb_used_meta":0,"kb_avail":280390764,"statfs":{"total":287844495360,"available":287120142336,"internally_reserved":16384,"allocated":287120158720,"data_stored":287120158720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":193544,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":14,"commit_latency_ns":0,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.129253+0000","last_change":"2021-04-04T09:29:44.129140+0000","last_active":"2021-04-04T09:29:44.129253+0000","last_peered":"2021-04-04T09:29:44.129253+0000","last_clean":"2021-04-04T09:29:44.129253+0000","last_became_active":"2021-04-04T09:29:44.128881+0000","last_became_peered":"2021-04-04T09:29:44.128881+0000","last_unstale":"2021-04-04T09:29:44.129253+0000","last_undegraded":"2021-04-04T09:29:44.129253+0000","last_fullsized":"2021-04-04T09:29:44.129253+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_clean_scrub_stamp":"2021-04-04T09:29:43.109397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.127841+0000","last_change":"2021-04-04T09:29:44.127325+0000","last_active":"2021-04-04T09:29:44.127841+0000","last_peered":"2021-04-04T09:29:44.127841+0000","last_clean":"2021-04-04T09:29:44.127841+0000","last_became_active":"2021-04-04T09:29:44.127024+0000","last_became_peered":"2021-04-04T09:29:44.127024+0000","last_unstale":"2021-04-04T09:29:44.127841+0000","last_undegraded":"2021-04-04T09:29:44.127841+0000","last_fullsized":"2021-04-04T09:29:44.127841+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_clean_scrub_stamp":"2021-04-04T09:29:43.109397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.129329+0000","last_change":"2021-04-04T09:29:44.129182+0000","last_active":"2021-04-04T09:29:44.129329+0000","last_peered":"2021-04-04T09:29:44.129329+0000","last_clean":"2021-04-04T09:29:44.129329+0000","last_became_active":"2021-04-04T09:29:44.128909+0000","last_became_peered":"2021-04-04T09:29:44.128909+0000","last_unstale":"2021-04-04T09:29:44.129329+0000","last_undegraded":"2021-04-04T09:29:44.129329+0000","last_fullsized":"2021-04-04T09:29:44.129329+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_clean_scrub_stamp":"2021-04-04T09:29:43.109397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.126727+0000","last_change":"2021-04-04T09:29:44.126513+0000","last_active":"2021-04-04T09:29:44.126727+0000","last_peered":"2021-04-04T09:29:44.126727+0000","last_clean":"2021-04-04T09:29:44.126727+0000","last_became_active":"2021-04-04T09:29:44.125787+0000","last_became_peered":"2021-04-04T09:29:44.125787+0000","last_unstale":"2021-04-04T09:29:44.126727+0000","last_undegraded":"2021-04-04T09:29:44.126727+0000","last_fullsized":"2021-04-04T09:29:44.126727+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_clean_scrub_stamp":"2021-04-04T09:29:43.109397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"11","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.125776+0000","last_change":"2021-04-04T09:29:40.236352+0000","last_active":"2021-04-04T09:29:44.125776+0000","last_peered":"2021-04-04T09:29:44.125776+0000","last_clean":"2021-04-04T09:29:44.125776+0000","last_became_active":"2021-04-04T09:29:40.235871+0000","last_became_peered":"2021-04-04T09:29:40.235871+0000","last_unstale":"2021-04-04T09:29:44.125776+0000","last_undegraded":"2021-04-04T09:29:44.125776+0000","last_fullsized":"2021-04-04T09:29:44.125776+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:39.079525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:39.079525+0000","last_clean_scrub_stamp":"2021-04-04T09:29:39.079525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.127861+0000","last_change":"2021-04-04T09:29:44.127491+0000","last_active":"2021-04-04T09:29:44.127861+0000","last_peered":"2021-04-04T09:29:44.127861+0000","last_clean":"2021-04-04T09:29:44.127861+0000","last_became_active":"2021-04-04T09:29:44.127302+0000","last_became_peered":"2021-04-04T09:29:44.127302+0000","last_unstale":"2021-04-04T09:29:44.127861+0000","last_undegraded":"2021-04-04T09:29:44.127861+0000","last_fullsized":"2021-04-04T09:29:44.127861+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_clean_scrub_stamp":"2021-04-04T09:29:43.109397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.128472+0000","last_change":"2021-04-04T09:29:44.127702+0000","last_active":"2021-04-04T09:29:44.128472+0000","last_peered":"2021-04-04T09:29:44.128472+0000","last_clean":"2021-04-04T09:29:44.128472+0000","last_became_active":"2021-04-04T09:29:44.127453+0000","last_became_peered":"2021-04-04T09:29:44.127453+0000","last_unstale":"2021-04-04T09:29:44.128472+0000","last_undegraded":"2021-04-04T09:29:44.128472+0000","last_fullsized":"2021-04-04T09:29:44.128472+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_clean_scrub_stamp":"2021-04-04T09:29:43.109397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.128883+0000","last_change":"2021-04-04T09:29:44.127962+0000","last_active":"2021-04-04T09:29:44.128883+0000","last_peered":"2021-04-04T09:29:44.128883+0000","last_clean":"2021-04-04T09:29:44.128883+0000","last_became_active":"2021-04-04T09:29:44.127461+0000","last_became_peered":"2021-04-04T09:29:44.127461+0000","last_unstale":"2021-04-04T09:29:44.128883+0000","last_undegraded":"2021-04-04T09:29:44.128883+0000","last_fullsized":"2021-04-04T09:29:44.128883+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_clean_scrub_stamp":"2021-04-04T09:29:43.109397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.127839+0000","last_change":"2021-04-04T09:29:44.127308+0000","last_active":"2021-04-04T09:29:44.127839+0000","last_peered":"2021-04-04T09:29:44.127839+0000","last_clean":"2021-04-04T09:29:44.127839+0000","last_became_active":"2021-04-04T09:29:44.127050+0000","last_became_peered":"2021-04-04T09:29:44.127050+0000","last_unstale":"2021-04-04T09:29:44.127839+0000","last_undegraded":"2021-04-04T09:29:44.127839+0000","last_fullsized":"2021-04-04T09:29:44.127839+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_clean_scrub_stamp":"2021-04-04T09:29:43.109397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235808,"kb_used_data":93463580,"kb_used_omap":48,"kb_used_meta":0,"kb_avail":93463572,"statfs":{"total":95948165120,"available":95706697728,"internally_reserved":8192,"allocated":95706705920,"data_stored":95706705920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":49312,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235760,"kb_used_data":93463620,"kb_used_omap":62,"kb_used_meta":0,"kb_avail":93463620,"statfs":{"total":95948165120,"available":95706746880,"internally_reserved":0,"allocated":95706746880,"data_stored":95706746880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64475,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":6,"commit_latency_ns":0,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235808,"kb_used_data":93463580,"kb_used_omap":77,"kb_used_meta":0,"kb_avail":93463572,"statfs":{"total":95948165120,"available":95706697728,"internally_reserved":8192,"allocated":95706705920,"data_stored":95706705920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79757,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":5,"commit_latency_ns":0,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]}]}],"pool_statfs":[]}} 2021-04-04T09:29:49.177 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2021-04-04T09:29:49.471 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:49.474 INFO:teuthology.orchestra.run.smithi122.stderr:dumped all 2021-04-04T09:29:49.485 INFO:teuthology.orchestra.run.smithi122.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-04-04T09:29:47.994189+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":281098140,"kb_used":707376,"kb_used_data":280390780,"kb_used_omap":189,"kb_used_meta":0,"kb_avail":280390764,"statfs":{"total":287844495360,"available":287120142336,"internally_reserved":16384,"allocated":287120158720,"data_stored":287120158720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":193544,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":14,"commit_latency_ns":0,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.129253+0000","last_change":"2021-04-04T09:29:44.129140+0000","last_active":"2021-04-04T09:29:44.129253+0000","last_peered":"2021-04-04T09:29:44.129253+0000","last_clean":"2021-04-04T09:29:44.129253+0000","last_became_active":"2021-04-04T09:29:44.128881+0000","last_became_peered":"2021-04-04T09:29:44.128881+0000","last_unstale":"2021-04-04T09:29:44.129253+0000","last_undegraded":"2021-04-04T09:29:44.129253+0000","last_fullsized":"2021-04-04T09:29:44.129253+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_clean_scrub_stamp":"2021-04-04T09:29:43.109397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.127841+0000","last_change":"2021-04-04T09:29:44.127325+0000","last_active":"2021-04-04T09:29:44.127841+0000","last_peered":"2021-04-04T09:29:44.127841+0000","last_clean":"2021-04-04T09:29:44.127841+0000","last_became_active":"2021-04-04T09:29:44.127024+0000","last_became_peered":"2021-04-04T09:29:44.127024+0000","last_unstale":"2021-04-04T09:29:44.127841+0000","last_undegraded":"2021-04-04T09:29:44.127841+0000","last_fullsized":"2021-04-04T09:29:44.127841+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_clean_scrub_stamp":"2021-04-04T09:29:43.109397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.129329+0000","last_change":"2021-04-04T09:29:44.129182+0000","last_active":"2021-04-04T09:29:44.129329+0000","last_peered":"2021-04-04T09:29:44.129329+0000","last_clean":"2021-04-04T09:29:44.129329+0000","last_became_active":"2021-04-04T09:29:44.128909+0000","last_became_peered":"2021-04-04T09:29:44.128909+0000","last_unstale":"2021-04-04T09:29:44.129329+0000","last_undegraded":"2021-04-04T09:29:44.129329+0000","last_fullsized":"2021-04-04T09:29:44.129329+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_clean_scrub_stamp":"2021-04-04T09:29:43.109397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.126727+0000","last_change":"2021-04-04T09:29:44.126513+0000","last_active":"2021-04-04T09:29:44.126727+0000","last_peered":"2021-04-04T09:29:44.126727+0000","last_clean":"2021-04-04T09:29:44.126727+0000","last_became_active":"2021-04-04T09:29:44.125787+0000","last_became_peered":"2021-04-04T09:29:44.125787+0000","last_unstale":"2021-04-04T09:29:44.126727+0000","last_undegraded":"2021-04-04T09:29:44.126727+0000","last_fullsized":"2021-04-04T09:29:44.126727+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_clean_scrub_stamp":"2021-04-04T09:29:43.109397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"11","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.125776+0000","last_change":"2021-04-04T09:29:40.236352+0000","last_active":"2021-04-04T09:29:44.125776+0000","last_peered":"2021-04-04T09:29:44.125776+0000","last_clean":"2021-04-04T09:29:44.125776+0000","last_became_active":"2021-04-04T09:29:40.235871+0000","last_became_peered":"2021-04-04T09:29:40.235871+0000","last_unstale":"2021-04-04T09:29:44.125776+0000","last_undegraded":"2021-04-04T09:29:44.125776+0000","last_fullsized":"2021-04-04T09:29:44.125776+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:39.079525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:39.079525+0000","last_clean_scrub_stamp":"2021-04-04T09:29:39.079525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.127861+0000","last_change":"2021-04-04T09:29:44.127491+0000","last_active":"2021-04-04T09:29:44.127861+0000","last_peered":"2021-04-04T09:29:44.127861+0000","last_clean":"2021-04-04T09:29:44.127861+0000","last_became_active":"2021-04-04T09:29:44.127302+0000","last_became_peered":"2021-04-04T09:29:44.127302+0000","last_unstale":"2021-04-04T09:29:44.127861+0000","last_undegraded":"2021-04-04T09:29:44.127861+0000","last_fullsized":"2021-04-04T09:29:44.127861+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_clean_scrub_stamp":"2021-04-04T09:29:43.109397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.128472+0000","last_change":"2021-04-04T09:29:44.127702+0000","last_active":"2021-04-04T09:29:44.128472+0000","last_peered":"2021-04-04T09:29:44.128472+0000","last_clean":"2021-04-04T09:29:44.128472+0000","last_became_active":"2021-04-04T09:29:44.127453+0000","last_became_peered":"2021-04-04T09:29:44.127453+0000","last_unstale":"2021-04-04T09:29:44.128472+0000","last_undegraded":"2021-04-04T09:29:44.128472+0000","last_fullsized":"2021-04-04T09:29:44.128472+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_clean_scrub_stamp":"2021-04-04T09:29:43.109397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.128883+0000","last_change":"2021-04-04T09:29:44.127962+0000","last_active":"2021-04-04T09:29:44.128883+0000","last_peered":"2021-04-04T09:29:44.128883+0000","last_clean":"2021-04-04T09:29:44.128883+0000","last_became_active":"2021-04-04T09:29:44.127461+0000","last_became_peered":"2021-04-04T09:29:44.127461+0000","last_unstale":"2021-04-04T09:29:44.128883+0000","last_undegraded":"2021-04-04T09:29:44.128883+0000","last_fullsized":"2021-04-04T09:29:44.128883+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_clean_scrub_stamp":"2021-04-04T09:29:43.109397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-04-04T09:29:44.127839+0000","last_change":"2021-04-04T09:29:44.127308+0000","last_active":"2021-04-04T09:29:44.127839+0000","last_peered":"2021-04-04T09:29:44.127839+0000","last_clean":"2021-04-04T09:29:44.127839+0000","last_became_active":"2021-04-04T09:29:44.127050+0000","last_became_peered":"2021-04-04T09:29:44.127050+0000","last_unstale":"2021-04-04T09:29:44.127839+0000","last_undegraded":"2021-04-04T09:29:44.127839+0000","last_fullsized":"2021-04-04T09:29:44.127839+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-04-04T09:29:43.109397+0000","last_clean_scrub_stamp":"2021-04-04T09:29:43.109397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235808,"kb_used_data":93463580,"kb_used_omap":48,"kb_used_meta":0,"kb_avail":93463572,"statfs":{"total":95948165120,"available":95706697728,"internally_reserved":8192,"allocated":95706705920,"data_stored":95706705920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":49312,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235760,"kb_used_data":93463620,"kb_used_omap":62,"kb_used_meta":0,"kb_avail":93463620,"statfs":{"total":95948165120,"available":95706746880,"internally_reserved":0,"allocated":95706746880,"data_stored":95706746880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64475,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":6,"commit_latency_ns":0,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":235808,"kb_used_data":93463580,"kb_used_omap":77,"kb_used_meta":0,"kb_avail":93463572,"statfs":{"total":95948165120,"available":95706697728,"internally_reserved":8192,"allocated":95706705920,"data_stored":95706705920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79757,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":5,"commit_latency_ns":0,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]}]}],"pool_statfs":[]}} 2021-04-04T09:29:49.486 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2021-04-04T09:29:49.487 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2021-04-04T09:29:49.487 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2021-04-04T09:29:49.487 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2021-04-04T09:29:49.838 INFO:teuthology.orchestra.run.smithi122.stdout: 2021-04-04T09:29:49.852 INFO:teuthology.orchestra.run.smithi122.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-04-04T09:29:49.853 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2021-04-04T09:29:49.853 INFO:teuthology.run_tasks:Running task exec... 2021-04-04T09:29:49.866 INFO:teuthology.task.exec:Executing custom commands... 2021-04-04T09:29:49.867 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi122.front.sepia.ceph.com 2021-04-04T09:29:49.868 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-04-04T09:29:50.252 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.4 2021-04-04T09:29:50.266 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-04-04T09:29:50.646 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.5 2021-04-04T09:29:50.659 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-04-04T09:29:51.037 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.6 2021-04-04T09:29:51.051 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-04-04T09:29:51.421 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.4 2021-04-04T09:29:51.434 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-04-04T09:29:51.817 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.5 2021-04-04T09:29:51.830 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-04-04T09:29:52.206 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.6 2021-04-04T09:29:52.219 INFO:teuthology.run_tasks:Running task exec... 2021-04-04T09:29:52.231 INFO:teuthology.task.exec:Executing custom commands... 2021-04-04T09:29:52.232 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi122.front.sepia.ceph.com 2021-04-04T09:29:52.232 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2021-04-04T09:29:52.618 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror 2021-04-04T09:29:52.631 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-04-04T09:29:53.014 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.0 2021-04-04T09:29:53.030 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-04-04T09:29:53.411 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.1 2021-04-04T09:29:53.425 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-04-04T09:29:53.779 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.2 2021-04-04T09:29:53.792 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-04-04T09:29:54.152 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.3 2021-04-04T09:29:54.164 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2021-04-04T09:29:54.533 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror 2021-04-04T09:29:54.552 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-04-04T09:29:54.901 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.0 2021-04-04T09:29:54.913 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-04-04T09:29:55.268 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.1 2021-04-04T09:29:55.281 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-04-04T09:29:55.657 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.2 2021-04-04T09:29:55.669 DEBUG:teuthology.orchestra.run.smithi122:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2021-04-04T09:29:56.043 INFO:teuthology.orchestra.run.smithi122.stderr:updated caps for client.mirror.3 2021-04-04T09:29:56.054 INFO:teuthology.run_tasks:Running task workunit... 2021-04-04T09:29:56.068 INFO:tasks.workunit:Pulling workunits from ref f4fb41419a42bee43a2069637ef17064073bf30b 2021-04-04T09:29:56.069 INFO:tasks.workunit:Making a separate scratch dir for every client... 2021-04-04T09:29:56.070 DEBUG:teuthology.orchestra.run.smithi122:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2021-04-04T09:29:56.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-04-04T09:29:56.077 INFO:teuthology.orchestra.run.smithi122.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2021-04-04T09:29:56.078 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2021-04-04T09:29:56.126 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2021-04-04T09:29:56.126 DEBUG:teuthology.orchestra.run.smithi122:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2021-04-04T09:29:56.174 INFO:tasks.workunit:timeout=3h 2021-04-04T09:29:56.174 INFO:tasks.workunit:cleanup=True 2021-04-04T09:29:56.175 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout f4fb41419a42bee43a2069637ef17064073bf30b 2021-04-04T09:29:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2021-04-04T09:30:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Note: checking out 'f4fb41419a42bee43a2069637ef17064073bf30b'. 2021-04-04T09:30:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2021-04-04T09:30:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2021-04-04T09:30:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:changes and commit them, and you can discard any commits you make in this 2021-04-04T09:30:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:state without impacting any branches by performing another checkout. 2021-04-04T09:30:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2021-04-04T09:30:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:If you want to create a new branch to retain commits you create, you may 2021-04-04T09:30:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:do so (now or later) by using -b with the checkout command again. Example: 2021-04-04T09:30:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2021-04-04T09:30:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: git checkout -b 2021-04-04T09:30:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2021-04-04T09:30:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:HEAD is now at f4fb41419a Merge PR #39488 into pacific 2021-04-04T09:30:32.640 DEBUG:teuthology.orchestra.run.smithi122:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2021-04-04T09:30:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2021-04-04T09:30:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2021-04-04T09:30:32.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2021-04-04T09:30:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2021-04-04T09:30:33.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2021-04-04T09:30:33.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2021-04-04T09:30:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2021-04-04T09:30:33.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2021-04-04T09:30:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2021-04-04T09:30:33.680 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2021-04-04T09:30:33.680 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2021-04-04T09:30:33.727 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_journal.sh on cluster1.client.mirror... 2021-04-04T09:30:33.728 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_journal.sh... 2021-04-04T09:30:33.729 DEBUG:teuthology.orchestra.run.smithi122:workunit test rbd/rbd_mirror_journal.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f4fb41419a42bee43a2069637ef17064073bf30b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ARGS='' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh 2021-04-04T09:30:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh 2021-04-04T09:30:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2021-04-04T09:30:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ type xmlstarlet 2021-04-04T09:30:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ XMLSTARLET=xmlstarlet 2021-04-04T09:30:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ RBD_MIRROR_INSTANCES=4 2021-04-04T09:30:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CLUSTER1=cluster1 2021-04-04T09:30:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CLUSTER2=cluster2 2021-04-04T09:30:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ PEER_CLUSTER_SUFFIX= 2021-04-04T09:30:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ POOL=mirror 2021-04-04T09:30:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ PARENT_POOL=mirror_parent 2021-04-04T09:30:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ NS1=ns1 2021-04-04T09:30:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ NS2=ns2 2021-04-04T09:30:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ TEMPDIR= 2021-04-04T09:30:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ID=mirror 2021-04-04T09:30:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ RBD_IMAGE_FEATURES=layering,exclusive-lock,journaling 2021-04-04T09:30:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ MIRROR_USER_ID_PREFIX=mirror. 2021-04-04T09:30:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ MIRROR_POOL_MODE=pool 2021-04-04T09:30:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ MIRROR_IMAGE_MODE=journal 2021-04-04T09:30:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ export CEPH_ARGS=--id mirror 2021-04-04T09:30:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ LAST_MIRROR_INSTANCE=3 2021-04-04T09:30:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh 2021-04-04T09:30:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2021-04-04T09:30:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2021-04-04T09:30:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_BIN=. 2021-04-04T09:30:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_SRC=. 2021-04-04T09:30:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -e CMakeCache.txt ] 2021-04-04T09:30:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 -gt 0 ] 2021-04-04T09:30:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ setup 2021-04-04T09:30:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local c 2021-04-04T09:30:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ trap cleanup $? INT TERM EXIT 2021-04-04T09:30:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ setup_tempdir 2021-04-04T09:30:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:30:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mktemp -d 2021-04-04T09:30:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ TEMPDIR=/tmp/tmp.zj6T3cKcnO 2021-04-04T09:30:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -z 1 ] 2021-04-04T09:30:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ setup_pools cluster1 cluster2 2021-04-04T09:30:33.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local remote_cluster=cluster2 2021-04-04T09:30:33.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mon_map_file 2021-04-04T09:30:33.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mon_addr 2021-04-04T09:30:33.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local admin_key_file 2021-04-04T09:30:33.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local uuid 2021-04-04T09:30:33.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool create mirror 64 64 2021-04-04T09:30:34.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:pool 'mirror' already exists 2021-04-04T09:30:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool create mirror_parent 64 64 2021-04-04T09:30:35.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:pool 'mirror_parent' already exists 2021-04-04T09:30:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 pool init mirror 2021-04-04T09:30:38.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 pool init mirror_parent 2021-04-04T09:30:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:30:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror pool 2021-04-04T09:30:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2021-04-04T09:30:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2021-04-04T09:30:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2021-04-04T09:30:41.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 pool 2021-04-04T09:30:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2021-04-04T09:30:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -z ] 2021-04-04T09:30:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -z ] 2021-04-04T09:30:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_add cluster1 mirror cluster2 2021-04-04T09:30:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local client_cluster=cluster2 2021-04-04T09:30:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local remote_cluster=cluster2 2021-04-04T09:30:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local uuid_var_name 2021-04-04T09:30:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:30:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local error_code 2021-04-04T09:30:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local peer_uuid 2021-04-04T09:30:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2021-04-04T09:30:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2021-04-04T09:30:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_uuid=ba5e7ea1-e930-4550-b570-945e322d5ab4 2021-04-04T09:30:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=0 2021-04-04T09:30:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2021-04-04T09:30:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 -eq 17 ] 2021-04-04T09:30:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 0 -eq 0 2021-04-04T09:30:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:30:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:30:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_add cluster1 mirror_parent cluster2 2021-04-04T09:30:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:30:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local client_cluster=cluster2 2021-04-04T09:30:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:42.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local remote_cluster=cluster2 2021-04-04T09:30:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local uuid_var_name 2021-04-04T09:30:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:30:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local error_code 2021-04-04T09:30:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local peer_uuid 2021-04-04T09:30:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2021-04-04T09:30:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2021-04-04T09:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_uuid=062cbc91-4713-4ded-b744-6fda24ecf93a 2021-04-04T09:30:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=0 2021-04-04T09:30:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2021-04-04T09:30:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 -eq 17 ] 2021-04-04T09:30:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 0 -eq 0 2021-04-04T09:30:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:30:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:30:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ setup_pools cluster2 cluster1 2021-04-04T09:30:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local remote_cluster=cluster1 2021-04-04T09:30:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mon_map_file 2021-04-04T09:30:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mon_addr 2021-04-04T09:30:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local admin_key_file 2021-04-04T09:30:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local uuid 2021-04-04T09:30:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool create mirror 64 64 2021-04-04T09:30:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:pool 'mirror' already exists 2021-04-04T09:30:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool create mirror_parent 64 64 2021-04-04T09:30:44.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:pool 'mirror_parent' already exists 2021-04-04T09:30:44.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 pool init mirror 2021-04-04T09:30:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 pool init mirror_parent 2021-04-04T09:30:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:30:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror pool 2021-04-04T09:30:50.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2021-04-04T09:30:50.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2021-04-04T09:30:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2021-04-04T09:30:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 pool 2021-04-04T09:30:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2021-04-04T09:30:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -z ] 2021-04-04T09:30:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -z ] 2021-04-04T09:30:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_add cluster2 mirror cluster1 2021-04-04T09:30:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local client_cluster=cluster1 2021-04-04T09:30:50.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local remote_cluster=cluster1 2021-04-04T09:30:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local uuid_var_name 2021-04-04T09:30:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:30:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local error_code 2021-04-04T09:30:50.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local peer_uuid 2021-04-04T09:30:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2021-04-04T09:30:50.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2021-04-04T09:30:50.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_uuid=ba5c4a7e-49df-4b8d-99f2-b5377df755f6 2021-04-04T09:30:50.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=0 2021-04-04T09:30:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2021-04-04T09:30:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 -eq 17 ] 2021-04-04T09:30:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 0 -eq 0 2021-04-04T09:30:50.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:30:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:30:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_add cluster2 mirror_parent cluster1 2021-04-04T09:30:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:50.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:30:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local client_cluster=cluster1 2021-04-04T09:30:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:50.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local remote_cluster=cluster1 2021-04-04T09:30:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local uuid_var_name 2021-04-04T09:30:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:30:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local error_code 2021-04-04T09:30:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local peer_uuid 2021-04-04T09:30:50.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2021-04-04T09:30:50.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2021-04-04T09:30:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_uuid=66f7b743-d0ed-4180-9599-aa4c8b596b8d 2021-04-04T09:30:50.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=0 2021-04-04T09:30:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2021-04-04T09:30:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 -eq 17 ] 2021-04-04T09:30:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 0 -eq 0 2021-04-04T09:30:50.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:30:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:30:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:30:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog TEST: add image and test replay 2021-04-04T09:30:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + date +%F %T 2021-04-04T09:30:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:30:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:30:50 TEST: add image and test replay 2021-04-04T09:30:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:30:50 TEST: add image and test replay 2021-04-04T09:30:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirrors cluster1 2021-04-04T09:30:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:30:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster1:0 2021-04-04T09:30:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:30:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:30:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:30:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:30:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:50.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:30:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:30:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:30:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:30:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:30:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster1:1 2021-04-04T09:30:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:30:50.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:30:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:30:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:30:50.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:30:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:30:50.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:30:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:30:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:30:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:30:50.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:30:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster1:2 2021-04-04T09:30:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:30:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:30:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:30:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:30:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:30:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:30:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:30:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:30:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:30:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:30:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:30:50.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster1:3 2021-04-04T09:30:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:30:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:50.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:30:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:30:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:30:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:50.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:30:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:30:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:30:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:50.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:30:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:30:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:30:50.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:30:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image=test 2021-04-04T09:30:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror test 2021-04-04T09:30:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:50.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2021-04-04T09:30:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:30:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test 2021-04-04T09:30:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_image_meta cluster2 mirror test key1 value1 2021-04-04T09:30:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:50.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local key=key1 2021-04-04T09:30:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=value1 2021-04-04T09:30:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 image-meta set mirror/test key1 value1 2021-04-04T09:30:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_image_meta cluster2 mirror test key2 value2 2021-04-04T09:30:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local key=key2 2021-04-04T09:30:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=value2 2021-04-04T09:30:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 image-meta set mirror/test key2 value2 2021-04-04T09:30:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-04-04T09:30:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-04-04T09:30:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:30:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:30:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:30:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2021-04-04T09:30:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:30:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:30:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:30:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:30:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:30:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:30:51.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:30:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + sleep 0 2021-04-04T09:30:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:30:51.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:30:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:30:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:30:51.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:30:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:30:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:30:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:30:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:51.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:30:51.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:30:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:30:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:30:51.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:30:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:30:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:30:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:30:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:30:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:30:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:30:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:30:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:30:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:30:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:30:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:30:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:30:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:30:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:30:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:30:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:30:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:30:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:51.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:30:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:30:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:30:51.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:30:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:30:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:30:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:30:51.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:30:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:30:51.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:30:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:30:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:30:51.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:30:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:30:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:30:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:30:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:30:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:30:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:30:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:30:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:30:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:30:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:30:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:30:52.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-04-04T09:30:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:30:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:30:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:30:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:30:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:30:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:30:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:30:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:30:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:30:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:30:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:30:52.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:30:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:30:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:30:52.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:30:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:30:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:30:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:30:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:30:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:30:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:30:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:30:52.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:30:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:30:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-04-04T09:30:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:30:52.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:30:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:30:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2021-04-04T09:30:52.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:30:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:52.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:30:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:30:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:30:52.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:30:52.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:30:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:30:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:30:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:30:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:30:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:30:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:30:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:30:52.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:30:52.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:52.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:52.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:30:52.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:30:52.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:30:52.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:30:52.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:30:52.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:30:52.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:30:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:30:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:30:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:30:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2021-04-04T09:30:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:30:52.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:30:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:30:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:30:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:30:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:30:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:30:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:30:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:53.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:30:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:30:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:30:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:30:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:30:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:53.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:30:53.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:53.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:30:53.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:30:53.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:30:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:30:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:30:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:30:53.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:30:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:30:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:30:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:53.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:30:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:30:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:30:53.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:30:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:30:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:30:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:53.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:30:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:30:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:30:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:53.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:30:53.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:30:53.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:30:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:30:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:30:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:30:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:30:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:30:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:30:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:30:53.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:53.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:53.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:30:53.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:30:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:30:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:30:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:30:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:30:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:30:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:30:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:30:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:30:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:30:53.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:30:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:30:53.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:30:53.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-04-04T09:30:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:30:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:30:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:30:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:30:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:30:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:30:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:30:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:30:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:30:53.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:30:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:30:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:30:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:30:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:30:53.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:30:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:30:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:30:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:30:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:30:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:30:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:30:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:30:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:30:53.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:30:53.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-04-04T09:30:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:30:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:30:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:30:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2021-04-04T09:30:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:30:53.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:30:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:30:53.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:30:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:30:53.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:30:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:30:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:30:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:30:53.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:30:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:30:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:53.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:30:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:30:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:30:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:53.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:30:53.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:30:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:30:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:30:53.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:30:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:30:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:30:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:30:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2021-04-04T09:30:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:30:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:30:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:30:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:30:55.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:30:55.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:30:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:30:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:30:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:30:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:30:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:30:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:55.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:30:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:30:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:30:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:30:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:30:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:30:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:30:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:30:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:30:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:30:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:30:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:30:55.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:55.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:30:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:30:55.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:30:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:30:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:30:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:30:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:30:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:30:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:30:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:30:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test 100 2021-04-04T09:30:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:55.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:30:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:30:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:30:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:30:55.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:30:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:30:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:30:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2500 bytes/sec: 9.8 MiB/s 2021-04-04T09:30:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2021-04-04T09:30:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:30:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:30:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2021-04-04T09:30:56.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:30:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:30:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:30:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:30:56.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:30:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test 2021-04-04T09:30:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:30:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:30:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test 2021-04-04T09:30:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2021-04-04T09:30:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:30:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:30:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:30:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:30:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:30:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:30:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2021-04-04T09:30:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:30:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:30:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:30:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:30:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:30:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:30:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:30:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:56.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:30:56.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:30:56.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:56.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:56.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:30:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:30:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:30:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:30:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:30:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:30:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:30:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:30:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2021-04-04T09:30:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:30:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test 2021-04-04T09:30:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test 2021-04-04T09:30:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:30:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:30:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:30:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:30:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:30:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:30:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:30:56.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:30:56.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2021-04-04T09:30:56.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2021-04-04T09:30:56.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=4, entry_tid=101 2021-04-04T09:30:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test 2021-04-04T09:30:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test ..* 2021-04-04T09:30:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:30:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:30:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:30:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:30:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:30:56.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:30:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:30:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:30:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2021-04-04T09:30:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=3, entry_tid=3], [object_number=2, tag_tid=3, entry_tid=2], [object_number=1, tag_tid=3, entry_tid=1], [object_number=0, tag_tid=3, entry_tid=0]]], state=connected] 2021-04-04T09:30:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=3, tag_tid=3, entry_tid=3 2021-04-04T09:30:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=4, entry_tid=101 -a object_number=1, tag_tid=4, entry_tid=101 = object_number=3, tag_tid=3, entry_tid=3 2021-04-04T09:30:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=3, tag_tid=3, entry_tid=3 != 2021-04-04T09:30:56.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:30:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=3, tag_tid=3, entry_tid=3 = 2021-04-04T09:30:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_mirror_pos=object_number=3, tag_tid=3, entry_tid=3 2021-04-04T09:30:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=4, entry_tid=101 2021-04-04T09:30:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo tag_tid=[0-9]* 2021-04-04T09:30:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:30:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_tag=4 2021-04-04T09:30:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=3, tag_tid=3, entry_tid=3 2021-04-04T09:30:56.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo tag_tid=[0-9]* 2021-04-04T09:30:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:30:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_tag=3 2021-04-04T09:30:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=4, entry_tid=101 2021-04-04T09:30:56.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo entry_tid=[0-9]* 2021-04-04T09:30:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:30:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_entry=101 2021-04-04T09:30:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=3, tag_tid=3, entry_tid=3 2021-04-04T09:30:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo entry_tid=[0-9]* 2021-04-04T09:30:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:30:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_entry=3 2021-04-04T09:30:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 4 = 3 -a 101 -le 3 2021-04-04T09:30:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:30:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:30:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test 2021-04-04T09:30:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:30:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:30:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test 2021-04-04T09:30:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2021-04-04T09:30:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:30:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:30:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:30:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:30:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:30:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:30:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2021-04-04T09:30:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:30:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:30:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:30:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:30:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:56.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:30:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:30:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:30:56.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:30:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:30:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:30:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:30:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:30:56.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:30:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:30:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:30:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:30:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:56.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:30:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:56.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2021-04-04T09:30:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:30:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test 2021-04-04T09:30:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test 2021-04-04T09:30:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:30:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:30:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:30:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:30:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:30:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:30:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:30:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:30:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2021-04-04T09:30:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=81], [object_number=0, tag_tid=4, entry_tid=80], [object_number=3, tag_tid=4, entry_tid=79], [object_number=2, tag_tid=4, entry_tid=78]]], state=connected] 2021-04-04T09:30:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=4, entry_tid=101 2021-04-04T09:30:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test 2021-04-04T09:30:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test ..* 2021-04-04T09:30:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:30:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:30:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:30:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:30:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:30:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:30:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:30:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:30:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2021-04-04T09:30:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=81], [object_number=0, tag_tid=4, entry_tid=80], [object_number=3, tag_tid=4, entry_tid=79], [object_number=2, tag_tid=4, entry_tid=78]]], state=connected] 2021-04-04T09:30:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:56.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=1, tag_tid=4, entry_tid=81 2021-04-04T09:30:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=4, entry_tid=101 -a object_number=1, tag_tid=4, entry_tid=101 = object_number=1, tag_tid=4, entry_tid=81 2021-04-04T09:30:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=1, tag_tid=4, entry_tid=81 != object_number=3, tag_tid=3, entry_tid=3 2021-04-04T09:30:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:30:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=1, tag_tid=4, entry_tid=81 = object_number=3, tag_tid=3, entry_tid=3 2021-04-04T09:30:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_mirror_pos=object_number=1, tag_tid=4, entry_tid=81 2021-04-04T09:30:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=4, entry_tid=101 2021-04-04T09:30:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo tag_tid=[0-9]* 2021-04-04T09:30:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:30:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_tag=4 2021-04-04T09:30:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=4, entry_tid=81 2021-04-04T09:30:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo tag_tid=[0-9]* 2021-04-04T09:30:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:30:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_tag=4 2021-04-04T09:30:56.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=4, entry_tid=101 2021-04-04T09:30:56.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo entry_tid=[0-9]* 2021-04-04T09:30:56.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:30:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_entry=101 2021-04-04T09:30:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=4, entry_tid=81 2021-04-04T09:30:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo entry_tid=[0-9]* 2021-04-04T09:30:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:30:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_entry=81 2021-04-04T09:30:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 4 = 4 -a 101 -le 81 2021-04-04T09:30:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:30:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:30:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test 2021-04-04T09:30:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:30:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:30:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test 2021-04-04T09:30:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2021-04-04T09:30:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:30:57.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:30:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:30:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:30:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:30:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:30:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2021-04-04T09:30:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:30:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:30:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:30:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:30:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:30:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:30:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:30:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:30:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:30:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:30:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:30:57.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:30:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:30:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:30:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:30:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:30:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:30:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:30:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:30:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:30:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:30:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:30:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:30:57.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2021-04-04T09:30:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:30:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test 2021-04-04T09:30:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test 2021-04-04T09:30:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:57.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:30:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:30:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + echo cluster2-mirror-test.status 2021-04-04T09:30:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:sed -e s|[/ ]|_|g 2021-04-04T09:30:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:30:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:30:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:30:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:30:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2021-04-04T09:30:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2021-04-04T09:30:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=4, entry_tid=101 2021-04-04T09:30:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test 2021-04-04T09:30:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test ..* 2021-04-04T09:30:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:30:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:30:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:30:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:30:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:30:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:30:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:30:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:30:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2021-04-04T09:30:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=101], [object_number=0, tag_tid=4, entry_tid=100], [object_number=3, tag_tid=4, entry_tid=99], [object_number=2, tag_tid=4, entry_tid=98]]], state=connected] 2021-04-04T09:30:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:30:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=1, tag_tid=4, entry_tid=101 2021-04-04T09:30:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=4, entry_tid=101 -a object_number=1, tag_tid=4, entry_tid=101 = object_number=1, tag_tid=4, entry_tid=101 2021-04-04T09:30:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:30:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2021-04-04T09:30:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:30:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:30:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:30:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:30:58.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2021-04-04T09:30:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:30:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:30:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:30:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:30:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:30:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:30:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:30:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:30:58.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:30:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:30:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T09:30:58.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:30:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2021-04-04T09:30:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2021-04-04T09:30:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:30:53 2021-04-04T09:30:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:30:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:30:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .*primary_position /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:30:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2021-04-04T09:30:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:30:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:30:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:30:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:30:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:30:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:30:58.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:30:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:30:53 2021-04-04T09:30:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2021-04-04 09:30:53 2021-04-04T09:30:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:30:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:58.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:30:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:30:53 2021-04-04T09:30:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:30:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:30:58.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:30:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:30:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:30:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:30:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:30:58.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:30:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:30:53 2021-04-04T09:30:58.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:30:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:30:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:30:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:30:58.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:30:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:30:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:30:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -z ] 2021-04-04T09:30:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test down+unknown 2021-04-04T09:30:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=down+unknown 2021-04-04T09:30:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:30:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:30:58.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:30:59.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster2 mirror test down+unknown 2021-04-04T09:30:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:59.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=down+unknown 2021-04-04T09:30:59.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:30:59.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:30:59.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.mirror_status 2021-04-04T09:30:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.mirror_status 2021-04-04T09:30:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:30:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:30:59.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror image status mirror/test 2021-04-04T09:30:59.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:30:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:30:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T09:30:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: down+unknown 2021-04-04T09:30:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: status not found 2021-04-04T09:30:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04T09:30:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:30:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster1 2021-04-04T09:30:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:30:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":3}} 2021-04-04T09:30:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:30:53 2021-04-04T09:30:59.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*down+unknown /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:30:59.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:30:59.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:30:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:30:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:30:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo down+unknown 2021-04-04T09:30:59.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:30:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:30:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:30:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update= 2021-04-04T09:30:59.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test down+unknown 2021-04-04T09:30:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:30:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=down+unknown 2021-04-04T09:30:59.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update= 2021-04-04T09:30:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror.mirror_status 2021-04-04T09:30:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror.mirror_status 2021-04-04T09:30:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:30:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror.mirror_status 2021-04-04T09:30:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2021-04-04T09:30:59.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:30:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:30:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:30:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update= 2021-04-04T09:30:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:30:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=down+unknown 2021-04-04T09:30:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo down+unknown 2021-04-04T09:30:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep down+unknown 2021-04-04T09:30:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:down+unknown 2021-04-04T09:30:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:30:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:30:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_images mirror test 2021-04-04T09:30:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:30:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:30:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2021-04-04T09:30:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.export 2021-04-04T09:30:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.export 2021-04-04T09:30:59.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:30:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export 2021-04-04T09:30:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.export 2021-04-04T09:30:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.export 2021-04-04T09:30:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:30:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:30:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:30:59.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export 2021-04-04T09:31:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:31:00.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:31:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:31:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:31:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:31:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_meta cluster1 mirror test key1 value1 2021-04-04T09:31:00.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local key=key1 2021-04-04T09:31:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local value=value1 2021-04-04T09:31:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 image-meta get mirror/test key1 2021-04-04T09:31:00.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test value1 = value1 2021-04-04T09:31:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_image_meta cluster1 mirror test key2 value2 2021-04-04T09:31:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:00.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local key=key2 2021-04-04T09:31:00.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local value=value2 2021-04-04T09:31:00.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 image-meta get mirror/test key2 2021-04-04T09:31:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test value2 = value2 2021-04-04T09:31:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog TEST: stop mirror, add image, start mirror and test replay 2021-04-04T09:31:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:31:00.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:31:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:31:00 TEST: stop mirror, add image, start mirror and test replay 2021-04-04T09:31:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:31:00 TEST: stop mirror, add image, start mirror and test replay 2021-04-04T09:31:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirrors cluster1 2021-04-04T09:31:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2021-04-04T09:31:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster1:0 2021-04-04T09:31:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2021-04-04T09:31:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:31:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid 2021-04-04T09:31:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:0 2021-04-04T09:31:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:00.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:00.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 pid file 2021-04-04T09:31:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2021-04-04T09:31:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2021-04-04T09:31:00.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pid=14582 2021-04-04T09:31:00.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n 14582 ] 2021-04-04T09:31:00.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ kill 14582 2021-04-04T09:31:00.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:31:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=14582 $2 == pid {print; exit 1} 2021-04-04T09:31:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:31:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:31:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=14582 $2 == pid {print; exit 1} 2021-04-04T09:31:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:01.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:01.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster2 2021-04-04T09:31:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:0 2021-04-04T09:31:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 pid file 2021-04-04T09:31:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2021-04-04T09:31:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2021-04-04T09:31:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster1:1 2021-04-04T09:31:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2021-04-04T09:31:01.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:31:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid 2021-04-04T09:31:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:1 2021-04-04T09:31:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 pid file 2021-04-04T09:31:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2021-04-04T09:31:01.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2021-04-04T09:31:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pid=14610 2021-04-04T09:31:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n 14610 ] 2021-04-04T09:31:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ kill 14610 2021-04-04T09:31:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:31:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=14610 $2 == pid {print; exit 1} 2021-04-04T09:31:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:31:02.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:31:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=14610 $2 == pid {print; exit 1} 2021-04-04T09:31:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:02.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:02.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster2 2021-04-04T09:31:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:02.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:02.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:02.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:02.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:02.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:1 2021-04-04T09:31:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:02.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 pid file 2021-04-04T09:31:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2021-04-04T09:31:02.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2021-04-04T09:31:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster1:2 2021-04-04T09:31:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:31:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2021-04-04T09:31:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:31:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid 2021-04-04T09:31:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:2 2021-04-04T09:31:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:31:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:31:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 pid file 2021-04-04T09:31:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2021-04-04T09:31:02.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2021-04-04T09:31:02.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pid=14638 2021-04-04T09:31:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n 14638 ] 2021-04-04T09:31:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ kill 14638 2021-04-04T09:31:02.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:31:03.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=14638 $2 == pid {print; exit 1} 2021-04-04T09:31:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:31:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:31:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=14638 $2 == pid {print; exit 1} 2021-04-04T09:31:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:31:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:31:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:31:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:31:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster2 2021-04-04T09:31:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:31:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:31:04.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:04.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:31:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:2 2021-04-04T09:31:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:31:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:31:04.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 pid file 2021-04-04T09:31:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2021-04-04T09:31:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2021-04-04T09:31:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster1:3 2021-04-04T09:31:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:31:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2021-04-04T09:31:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:31:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid 2021-04-04T09:31:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:3 2021-04-04T09:31:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:31:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:31:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:04.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 pid file 2021-04-04T09:31:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2021-04-04T09:31:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2021-04-04T09:31:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pid=14686 2021-04-04T09:31:04.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n 14686 ] 2021-04-04T09:31:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ kill 14686 2021-04-04T09:31:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:31:05.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=14686 $2 == pid {print; exit 1} 2021-04-04T09:31:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:31:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:31:05.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=14686 $2 == pid {print; exit 1} 2021-04-04T09:31:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:31:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:31:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:31:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:05.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:05.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:31:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster2 2021-04-04T09:31:05.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:31:05.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:05.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:05.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:31:05.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:05.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:31:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:3 2021-04-04T09:31:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:31:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:31:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:05.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:05.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 pid file 2021-04-04T09:31:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2021-04-04T09:31:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2021-04-04T09:31:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image1=test1 2021-04-04T09:31:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror test1 2021-04-04T09:31:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:05.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2021-04-04T09:31:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test1 2021-04-04T09:31:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test1 100 2021-04-04T09:31:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:31:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:31:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:31:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:31:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test1 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:31:05.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:31:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:31:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2777.78 bytes/sec: 11 MiB/s 2021-04-04T09:31:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirrors cluster1 2021-04-04T09:31:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:05.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster1:0 2021-04-04T09:31:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:05.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:05.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:05.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:31:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:31:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster1:1 2021-04-04T09:31:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:31:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:31:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster1:2 2021-04-04T09:31:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:31:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:31:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:31:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:31:05.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster1:3 2021-04-04T09:31:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:31:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:31:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:05.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:05.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:05.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:05.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:31:05.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:31:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2021-04-04T09:31:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2021-04-04T09:31:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:31:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test1 started 2021-04-04T09:31:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:31:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test1 2021-04-04T09:31:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:31:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i+ statelocal 2021-04-04T09:31:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: cluster=cluster1 2021-04-04T09:31:06.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test1 2021-04-04T09:31:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:06.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:06.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:06.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:06.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:06.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2021-04-04T09:31:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test1'] 2021-04-04T09:31:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:31:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test1 2021-04-04T09:31:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:06.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:06.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:06.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test1 2021-04-04T09:31:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test1'] 2021-04-04T09:31:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:31:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test1 2021-04-04T09:31:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:31:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:31:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:31:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:31:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:31:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:06.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:06.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:31:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:06.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test1 2021-04-04T09:31:06.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test1'] 2021-04-04T09:31:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:31:06.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test1 2021-04-04T09:31:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:31:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:31:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:31:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:31:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:31:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:31:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:06.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:06.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:06.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test1 2021-04-04T09:31:06.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test1'] 2021-04-04T09:31:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:31:06.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:07.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test1 2021-04-04T09:31:07.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:07.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:08.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2021-04-04T09:31:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test1'] 2021-04-04T09:31:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:31:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test1 2021-04-04T09:31:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:08.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:08.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:08.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:08.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:08.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test1 2021-04-04T09:31:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test1'] 2021-04-04T09:31:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:31:08.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test1 2021-04-04T09:31:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:31:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:31:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:08.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:31:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:31:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:08.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:31:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:08.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:31:08.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test1 2021-04-04T09:31:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test1'] 2021-04-04T09:31:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:31:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test1 2021-04-04T09:31:08.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:31:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:31:08.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:08.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:31:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:31:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:08.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:31:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:08.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:08.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:31:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:08.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test1 2021-04-04T09:31:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test1'] 2021-04-04T09:31:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:31:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:31:10.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:10.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:10.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test1 2021-04-04T09:31:10.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:10.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:10.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:10.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:10.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:10.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:10.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:10.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:10.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:10.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:10.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2021-04-04T09:31:10.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:10.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:31:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:31:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:31:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:31:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:31:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:31:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test1 2021-04-04T09:31:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:31:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:31:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test1 2021-04-04T09:31:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:31:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:31:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:31:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:31:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:31:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test1 2021-04-04T09:31:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:10.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:31:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test1 ] 2021-04-04T09:31:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test1 2021-04-04T09:31:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test1 2021-04-04T09:31:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:31:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test1 2021-04-04T09:31:10.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test1 2021-04-04T09:31:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test1 2021-04-04T09:31:10.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test1 2021-04-04T09:31:10.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:31:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test1.status 2021-04-04T09:31:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test1.status 2021-04-04T09:31:10.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.status 2021-04-04T09:31:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test1 2021-04-04T09:31:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.status 2021-04-04T09:31:11.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:31:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:31:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:31:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:31:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:31:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.status 2021-04-04T09:31:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=3, tag_tid=2, entry_tid=3 2021-04-04T09:31:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test1 2021-04-04T09:31:11.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test1 ..* 2021-04-04T09:31:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:31:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test1.status 2021-04-04T09:31:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test1.status 2021-04-04T09:31:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.status 2021-04-04T09:31:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.status 2021-04-04T09:31:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd --cluster cluster2 journal status --image mirror/test1 2021-04-04T09:31:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:31:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:31:11.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:31:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:31:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:31:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.status 2021-04-04T09:31:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=3, tag_tid=2, entry_tid=3 2021-04-04T09:31:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=3, tag_tid=2, entry_tid=3 -a object_number=3, tag_tid=2, entry_tid=3 = object_number=3, tag_tid=2, entry_tid=3 2021-04-04T09:31:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying primary_position 2021-04-04T09:31:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:11.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:11.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:11.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:11.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:31:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:12.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying primary_position 2021-04-04T09:31:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:12.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:31:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test1.mirror_status 2021-04-04T09:31:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test1.mirror_status 2021-04-04T09:31:12.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test1 2021-04-04T09:31:12.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test1: 2021-04-04T09:31:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:31:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:31:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:31:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2021-04-04T09:31:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:09 2021-04-04T09:31:12.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:31:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .*primary_position /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:12.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:31:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:12.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:12.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:31:12.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:12.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:09 2021-04-04T09:31:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying 2021-04-04 09:31:09 2021-04-04T09:31:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:12.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:12.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:09 2021-04-04T09:31:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:12.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:12.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:12.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/last_update 2021-04-04T09:31:12.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:09 2021-04-04T09:31:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/state 2021-04-04T09:31:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:31:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:31:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -z ] 2021-04-04T09:31:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test1 down+unknown 2021-04-04T09:31:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=down+unknown 2021-04-04T09:31:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster2 mirror test1 down+unknown 2021-04-04T09:31:13.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=down+unknown 2021-04-04T09:31:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:13.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test1.mirror_status 2021-04-04T09:31:13.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test1.mirror_status 2021-04-04T09:31:13.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.mirror_status 2021-04-04T09:31:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror image status mirror/test1 2021-04-04T09:31:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.mirror_status 2021-04-04T09:31:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test1: 2021-04-04T09:31:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:31:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: down+unknown 2021-04-04T09:31:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: status not found 2021-04-04T09:31:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04T09:31:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:31:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster1 2021-04-04T09:31:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:31:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:31:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:09 2021-04-04T09:31:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*down+unknown /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.mirror_status 2021-04-04T09:31:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:31:13.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.mirror_status 2021-04-04T09:31:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:31:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo down+unknown 2021-04-04T09:31:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.mirror_status 2021-04-04T09:31:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.mirror_status 2021-04-04T09:31:13.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update= 2021-04-04T09:31:13.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test1 down+unknown 2021-04-04T09:31:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=down+unknown 2021-04-04T09:31:13.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update= 2021-04-04T09:31:13.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror.mirror_status 2021-04-04T09:31:13.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror.mirror_status 2021-04-04T09:31:13.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:13.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror.mirror_status 2021-04-04T09:31:13.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/last_update 2021-04-04T09:31:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update= 2021-04-04T09:31:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/state 2021-04-04T09:31:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=down+unknown 2021-04-04T09:31:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo down+unknown 2021-04-04T09:31:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep down+unknown 2021-04-04T09:31:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:down+unknown 2021-04-04T09:31:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_images mirror test1 2021-04-04T09:31:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2021-04-04T09:31:13.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test1.export 2021-04-04T09:31:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test1.export 2021-04-04T09:31:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:13.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.export 2021-04-04T09:31:13.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test1.export 2021-04-04T09:31:13.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test1.export 2021-04-04T09:31:13.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:13.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.export 2021-04-04T09:31:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.export 2021-04-04T09:31:13.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 export mirror/test1 /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.export 2021-04-04T09:31:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:31:13.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 export mirror/test1 /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.export 2021-04-04T09:31:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:31:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.export 2021-04-04T09:31:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test1.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.export 2021-04-04T09:31:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog TEST: test the first image is replaying after restart 2021-04-04T09:31:13.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:31:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:31:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:31:13 TEST: test the first image is replaying after restart 2021-04-04T09:31:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:31:13 TEST: test the first image is replaying after restart 2021-04-04T09:31:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test 100 2021-04-04T09:31:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:31:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:31:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:31:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:31:13.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:31:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:31:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:31:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2500 bytes/sec: 9.8 MiB/s 2021-04-04T09:31:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-04-04T09:31:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-04-04T09:31:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:31:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:15.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2021-04-04T09:31:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:15.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:31:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:15.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:31:15.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + greplocal -i cluster_instance=cluster1 state 2021-04-04T09:31:15.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2021-04-04T09:31:15.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:15.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:15.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:15.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:15.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:31:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:15.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:15.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:15.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:31:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:31:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:31:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:15.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:31:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:15.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:15.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:15.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:15.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:15.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:15.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:15.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:15.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:31:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:31:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + grep -i Replaying 2021-04-04T09:31:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:echo "state": "Replaying" 2021-04-04T09:31:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:31:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:31:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:31:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2021-04-04T09:31:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:31:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:31:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2021-04-04T09:31:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:31:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:31:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:31:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:31:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:31:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test 2021-04-04T09:31:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:31:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:31:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test 2021-04-04T09:31:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2021-04-04T09:31:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:31:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2021-04-04T09:31:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:15.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:15.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:15.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:15.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2021-04-04T09:31:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:31:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush mirror/test1 2021-04-04T09:31:15.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2021-04-04T09:31:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:15.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:15.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:15.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:15.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:15.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2021-04-04T09:31:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test 2021-04-04T09:31:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test 2021-04-04T09:31:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:15.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:31:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:31:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:31:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:31:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:31:15.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:31:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:31:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:31:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:31:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T09:31:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T09:31:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:31:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=5, entry_tid=101 2021-04-04T09:31:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test 2021-04-04T09:31:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:15.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test ..* 2021-04-04T09:31:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:31:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:31:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:31:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:31:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:31:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:31:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:31:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:31:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:31:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:31:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T09:31:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T09:31:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:31:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=1, tag_tid=5, entry_tid=101 2021-04-04T09:31:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=5, entry_tid=101 -a object_number=1, tag_tid=5, entry_tid=101 = object_number=1, tag_tid=5, entry_tid=101 2021-04-04T09:31:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2021-04-04T09:31:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:31:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2021-04-04T09:31:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:31:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:31:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:31:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:31:16.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:31:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T09:31:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:31:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}} 2021-04-04T09:31:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:31:17.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:09 2021-04-04T09:31:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:31:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .*primary_position /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":4}} 2021-04-04T09:31:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:31:17.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:09 2021-04-04T09:31:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2021-04-04 09:31:09 2021-04-04T09:31:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:17.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:09 2021-04-04T09:31:17.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:17.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:31:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:09 2021-04-04T09:31:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:31:17.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:31:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:31:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:17.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_images mirror test 2021-04-04T09:31:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:17.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2021-04-04T09:31:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.export 2021-04-04T09:31:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.export 2021-04-04T09:31:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export 2021-04-04T09:31:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.export 2021-04-04T09:31:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.export 2021-04-04T09:31:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:17.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:31:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:31:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export 2021-04-04T09:31:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:31:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:31:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:31:17.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:31:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:31:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -z ] 2021-04-04T09:31:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog TEST: stop/start/restart mirror via admin socket 2021-04-04T09:31:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:31:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:31:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:31:17 TEST: stop/start/restart mirror via admin socket 2021-04-04T09:31:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:31:17 TEST: stop/start/restart mirror via admin socket 2021-04-04T09:31:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ all_admin_daemons cluster1 rbd mirror stop 2021-04-04T09:31:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror stop 2021-04-04T09:31:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:17.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:17.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:17.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop 2021-04-04T09:31:17.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror stop 2021-04-04T09:31:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:17.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:17.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:17.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop 2021-04-04T09:31:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror stop 2021-04-04T09:31:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:31:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:31:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:31:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:31:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:31:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:17.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:17.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:31:17.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror stop 2021-04-04T09:31:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror stop 2021-04-04T09:31:18.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:31:18.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:18.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:18.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:31:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:31:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:31:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:31:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror stop 2021-04-04T09:31:18.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2021-04-04T09:31:18.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:18.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2021-04-04T09:31:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=stopped 2021-04-04T09:31:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test stopped 2021-04-04T09:31:19.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:31:19.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:31:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:31:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:31:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:19.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:31:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:19.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:19.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:19.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:31:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:31:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:31:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:31:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:31:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:19.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:31:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Stopped" 2021-04-04T09:31:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:31:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = stopped 2021-04-04T09:31:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2021-04-04T09:31:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2021-04-04T09:31:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=stopped 2021-04-04T09:31:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2021-04-04T09:31:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:20.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:31:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1+ rbd mirror status mirror/test1 2021-04-04T09:31:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:31:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:31:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:20.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test1 2021-04-04T09:31:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:20.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:20.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:20.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2021-04-04T09:31:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:31:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Stopped" 2021-04-04T09:31:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:31:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = stopped 2021-04-04T09:31:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2021-04-04T09:31:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped 2021-04-04T09:31:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:31:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:31:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + CEPH_ARGS= rbd --cluster cluster1 mirrortee image /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:21.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: status mirror/test 2021-04-04T09:31:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:31:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T09:31:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:31:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: stopped 2021-04-04T09:31:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:31:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:17 2021-04-04T09:31:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+stopped /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:31:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: stopped 2021-04-04T09:31:21.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:31:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:21.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:31:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:21.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:31:21.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:17 2021-04-04T09:31:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped 2021-04-04 09:31:17 2021-04-04T09:31:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:21.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:21.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:17 2021-04-04T09:31:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:21.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:21.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:31:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:17 2021-04-04T09:31:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:31:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+stopped 2021-04-04T09:31:21.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:31:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+stopped 2021-04-04T09:31:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:31:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2021-04-04T09:31:21.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped 2021-04-04T09:31:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test1.mirror_status 2021-04-04T09:31:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test1.mirror_status 2021-04-04T09:31:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test1 2021-04-04T09:31:22.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test1: 2021-04-04T09:31:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:31:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:31:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: stopped 2021-04-04T09:31:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2021-04-04T09:31:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:17 2021-04-04T09:31:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+stopped /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:31:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: stopped 2021-04-04T09:31:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:31:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:31:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:31:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:17 2021-04-04T09:31:22.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped 2021-04-04 09:31:17 2021-04-04T09:31:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:22.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:22.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:17 2021-04-04T09:31:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:22.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:22.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:22.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/last_update 2021-04-04T09:31:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:17 2021-04-04T09:31:22.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/state 2021-04-04T09:31:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+stopped 2021-04-04T09:31:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:31:22.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+stopped 2021-04-04T09:31:22.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:31:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:22.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:22.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ all_admin_daemons cluster1 rbd mirror start 2021-04-04T09:31:22.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:22.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:22.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror start 2021-04-04T09:31:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:22.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:22.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:22.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:22.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:22.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start 2021-04-04T09:31:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror start 2021-04-04T09:31:23.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:23.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:23.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:23.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:23.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:23.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:23.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:23.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:23.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start 2021-04-04T09:31:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror start 2021-04-04T09:31:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:31:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:31:23.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:31:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:31:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:31:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:23.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:31:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror start 2021-04-04T09:31:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror start 2021-04-04T09:31:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:31:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:31:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:31:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:31:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:31:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:31:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:23.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror start 2021-04-04T09:31:23.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-04-04T09:31:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:23.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-04-04T09:31:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:31:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2021-04-04T09:31:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:31:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:31:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:31:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:31:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:31:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:31:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:31:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:31:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:31:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:31:24.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:31:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:31:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:31:24.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:31:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:31:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:31:24.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2021-04-04T09:31:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:24.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2021-04-04T09:31:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:24.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:31:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:24.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test1 started 2021-04-04T09:31:25.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:31:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test1 2021-04-04T09:31:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:31:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + shift 2021-04-04T09:31:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:31:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test1 2021-04-04T09:31:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:25.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:25.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2021-04-04T09:31:25.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:31:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:31:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:31:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:31:25.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:31:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:31:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2021-04-04T09:31:25.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:25.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2021-04-04T09:31:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:31:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:31:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:31:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:31:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T09:31:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:31:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T09:31:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:31:26.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:23 2021-04-04T09:31:26.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:31:26.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T09:31:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:31:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:23 2021-04-04T09:31:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2021-04-04 09:31:23 2021-04-04T09:31:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:23 2021-04-04T09:31:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:31:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:23 2021-04-04T09:31:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:31:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:31:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:31:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2021-04-04T09:31:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying 2021-04-04T09:31:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:28.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test1.mirror_status 2021-04-04T09:31:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test1.mirror_status 2021-04-04T09:31:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test1 2021-04-04T09:31:28.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test1: 2021-04-04T09:31:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:31:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:31:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:31:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2021-04-04T09:31:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:23 2021-04-04T09:31:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:31:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:31:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:28.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:31:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:23 2021-04-04T09:31:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying 2021-04-04 09:31:23 2021-04-04T09:31:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:23 2021-04-04T09:31:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/last_update 2021-04-04T09:31:28.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:23 2021-04-04T09:31:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/state 2021-04-04T09:31:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:31:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:31:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ all_admin_daemons cluster1 rbd mirror restart 2021-04-04T09:31:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror restart 2021-04-04T09:31:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:28.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart 2021-04-04T09:31:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror restart 2021-04-04T09:31:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart 2021-04-04T09:31:28.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror restart 2021-04-04T09:31:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:31:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:31:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:31:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:31:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:31:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:31:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror restart 2021-04-04T09:31:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror restart 2021-04-04T09:31:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:31:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:31:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:31:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:31:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:31:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:31:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror restart 2021-04-04T09:31:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-04-04T09:31:28.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:28.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-04-04T09:31:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:31:28.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:28.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2021-04-04T09:31:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:31:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:31:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i state 2021-04-04T09:31:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:31:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:29.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:29.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:31:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:29.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:29.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:29.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:29.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:29.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:29.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:31:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:31:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:31:29.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:31:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:29.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:29.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:29.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:29.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:29.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:31:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:31:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:31:30.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:31:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:31:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:31:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:31:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2021-04-04T09:31:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:30.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2021-04-04T09:31:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:31:30.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test1 started 2021-04-04T09:31:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:31:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test1 2021-04-04T09:31:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:31:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:31.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:31:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test1 2021-04-04T09:31:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:31.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:31.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:31.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:31.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:31.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:31.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:31.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2021-04-04T09:31:31.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:31.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:31:31.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:31:31.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:31:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:31:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:31:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:31:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2021-04-04T09:31:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2021-04-04T09:31:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:31:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:31:32.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:31:32.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:31:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T09:31:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:31:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T09:31:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:31:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:28 2021-04-04T09:31:32.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:32.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:31:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:32.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T09:31:32.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:32.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:31:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:28 2021-04-04T09:31:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2021-04-04 09:31:28 2021-04-04T09:31:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:28 2021-04-04T09:31:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:31:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:28 2021-04-04T09:31:32.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:31:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:31:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:31:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2021-04-04T09:31:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:33.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying 2021-04-04T09:31:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:33.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test1.mirror_status 2021-04-04T09:31:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test1.mirror_status 2021-04-04T09:31:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test1 2021-04-04T09:31:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test1: 2021-04-04T09:31:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:31:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:31:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:31:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2021-04-04T09:31:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:28 2021-04-04T09:31:33.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:33.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:31:33.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:33.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:31:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:33.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:33.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:33.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:31:33.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:28 2021-04-04T09:31:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying 2021-04-04 09:31:28 2021-04-04T09:31:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:28 2021-04-04T09:31:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:33.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:33.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:33.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/last_update 2021-04-04T09:31:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:28 2021-04-04T09:31:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/state 2021-04-04T09:31:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:31:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:31:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ all_admin_daemons cluster1 rbd mirror stop 2021-04-04T09:31:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror stop 2021-04-04T09:31:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:33.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:33.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:33.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:33.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:33.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop 2021-04-04T09:31:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror stop 2021-04-04T09:31:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:33.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:33.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:33.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:33.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:33.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop 2021-04-04T09:31:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror stop 2021-04-04T09:31:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:31:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:33.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:31:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:33.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:31:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:31:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:31:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:33.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:33.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:33.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:31:33.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:33.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror stop 2021-04-04T09:31:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror stop 2021-04-04T09:31:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:31:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:31:33.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:33.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:33.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:33.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:33.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:31:33.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:31:33.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:33.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:33.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:31:33.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:33.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:33.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:31:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror stop 2021-04-04T09:31:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2021-04-04T09:31:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:34.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2021-04-04T09:31:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=stopped 2021-04-04T09:31:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test stopped 2021-04-04T09:31:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:31:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:31:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:31:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster=cluster1 2021-04-04T09:31:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ greplocal -i loop_instance state 2021-04-04T09:31:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2021-04-04T09:31:35.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:31:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:35.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:35.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:31:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:31:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:31:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:31:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:35.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:31:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:31:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + echo "state": "Stopped"grep 2021-04-04T09:31:35.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: -i Replaying 2021-04-04T09:31:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = stopped 2021-04-04T09:31:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:35.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2021-04-04T09:31:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:35.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2021-04-04T09:31:35.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=stopped 2021-04-04T09:31:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2021-04-04T09:31:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:31:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test1 2021-04-04T09:31:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + greplocal -i cluster_instance=cluster1 state 2021-04-04T09:31:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2021-04-04T09:31:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test1 2021-04-04T09:31:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:36.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:36.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:36.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:36.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:36.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2021-04-04T09:31:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:31:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Stopped" 2021-04-04T09:31:36.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:31:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = stopped 2021-04-04T09:31:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2021-04-04T09:31:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped 2021-04-04T09:31:37.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:37.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:31:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:31:37.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:37.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:31:37.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:31:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T09:31:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:31:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: stopped 2021-04-04T09:31:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:31:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:33 2021-04-04T09:31:37.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+stopped /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:31:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:37.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: stopped 2021-04-04T09:31:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:31:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:31:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:31:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:33 2021-04-04T09:31:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped 2021-04-04 09:31:33 2021-04-04T09:31:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:33 2021-04-04T09:31:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:37.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:37.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:37.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:31:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:33 2021-04-04T09:31:37.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:31:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+stopped 2021-04-04T09:31:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:31:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+stopped 2021-04-04T09:31:37.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:31:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:37.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:37.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2021-04-04T09:31:37.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:37.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:37.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:37.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped 2021-04-04T09:31:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test1.mirror_status 2021-04-04T09:31:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test1.mirror_status 2021-04-04T09:31:38.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:38.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test1 2021-04-04T09:31:38.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test1: 2021-04-04T09:31:38.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:31:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:31:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: stopped 2021-04-04T09:31:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2021-04-04T09:31:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:33 2021-04-04T09:31:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+stopped /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:31:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:38.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: stopped 2021-04-04T09:31:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:31:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:38.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:31:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:38.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:31:38.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:38.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:33 2021-04-04T09:31:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped 2021-04-04 09:31:33 2021-04-04T09:31:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:33 2021-04-04T09:31:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/last_update 2021-04-04T09:31:38.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:33 2021-04-04T09:31:38.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/state 2021-04-04T09:31:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+stopped 2021-04-04T09:31:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:31:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+stopped 2021-04-04T09:31:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:31:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ all_admin_daemons cluster1 rbd mirror restart 2021-04-04T09:31:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror restart 2021-04-04T09:31:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:38.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:38.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:38.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:38.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart 2021-04-04T09:31:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror restart 2021-04-04T09:31:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:39.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart 2021-04-04T09:31:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror restart 2021-04-04T09:31:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:31:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:31:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:31:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:31:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:31:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:31:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror restart 2021-04-04T09:31:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror restart 2021-04-04T09:31:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:31:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:31:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:39.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:31:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:31:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:31:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:31:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror restart 2021-04-04T09:31:39.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-04-04T09:31:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-04-04T09:31:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:31:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2021-04-04T09:31:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:40.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:31:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:31:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:31:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local instance=cluster1 2021-04-04T09:31:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:31:40.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:31:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:40.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:40.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:31:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:31:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:31:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:31:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:40.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:40.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:31:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:31:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:31:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:31:40.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:31:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:31:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:31:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2021-04-04T09:31:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2021-04-04T09:31:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:31:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:41.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test1 started 2021-04-04T09:31:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:41.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:31:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test1 2021-04-04T09:31:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:31:41.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local instance=cluster1 2021-04-04T09:31:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:31:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test1 2021-04-04T09:31:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:41.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:41.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:41.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:41.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:41.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:41.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:41.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:41.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2021-04-04T09:31:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:31:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:31:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:31:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:31:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:31:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:31:41.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2021-04-04T09:31:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2021-04-04T09:31:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:31:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:31:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:31:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:42.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:31:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T09:31:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:31:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T09:31:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:31:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:39 2021-04-04T09:31:42.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:31:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T09:31:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:31:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:42.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:39 2021-04-04T09:31:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2021-04-04 09:31:39 2021-04-04T09:31:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:39 2021-04-04T09:31:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:42.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:42.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:42.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:42.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:31:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:39 2021-04-04T09:31:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:31:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:31:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:31:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2021-04-04T09:31:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:42.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:42.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:42.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying 2021-04-04T09:31:43.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test1.mirror_status 2021-04-04T09:31:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test1.mirror_status 2021-04-04T09:31:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:43.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test1 2021-04-04T09:31:43.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test1: 2021-04-04T09:31:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:31:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:31:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:31:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2021-04-04T09:31:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:39 2021-04-04T09:31:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:31:44.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:31:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:31:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:44.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:39 2021-04-04T09:31:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying 2021-04-04 09:31:39 2021-04-04T09:31:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:39 2021-04-04T09:31:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:44.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:44.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:44.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/last_update 2021-04-04T09:31:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:39 2021-04-04T09:31:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/state 2021-04-04T09:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:31:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:31:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ all_admin_daemons cluster1 rbd mirror stop mirror cluster2 2021-04-04T09:31:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror cluster2 2021-04-04T09:31:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:44.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:44.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:44.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:44.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror cluster2 2021-04-04T09:31:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror stop mirror cluster2 2021-04-04T09:31:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:44.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:44.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:44.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:44.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:44.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop mirror cluster2 2021-04-04T09:31:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror stop mirror cluster2 2021-04-04T09:31:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:31:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:31:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:31:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:31:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:31:44.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:44.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:44.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:31:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:44.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:44.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror stop mirror cluster2 2021-04-04T09:31:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror stop mirror cluster2 2021-04-04T09:31:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:31:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:44.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:31:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:44.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:44.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:44.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:44.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:44.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:31:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:31:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:31:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:44.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:44.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:44.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:44.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:44.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:31:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror stop mirror cluster2 2021-04-04T09:31:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2021-04-04T09:31:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2021-04-04T09:31:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:44.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:44.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:44.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=stopped 2021-04-04T09:31:44.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:44.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test stopped 2021-04-04T09:31:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:45.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:31:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:45.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:31:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:31:45.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local+ cluster=cluster1 2021-04-04T09:31:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ greplocal -i loop_instance state 2021-04-04T09:31:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2021-04-04T09:31:45.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:45.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:45.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:45.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:31:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:45.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:45.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:45.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:45.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:45.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:45.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:45.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:31:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:31:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:31:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:31:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:45.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:45.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:45.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:31:45.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:31:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Stopped" 2021-04-04T09:31:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:31:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = stopped 2021-04-04T09:31:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2021-04-04T09:31:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2021-04-04T09:31:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=stopped 2021-04-04T09:31:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2021-04-04T09:31:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:31:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:46.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test1 2021-04-04T09:31:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:31:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i+ state 2021-04-04T09:31:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:shift 2021-04-04T09:31:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:46.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test1 2021-04-04T09:31:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:46.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:46.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:46.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:46.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:46.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:46.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:46.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:46.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2021-04-04T09:31:47.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:31:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + echogrep -i "state": "Stopped" Replaying 2021-04-04T09:31:47.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2021-04-04T09:31:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = stopped 2021-04-04T09:31:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:47.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2021-04-04T09:31:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:47.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped 2021-04-04T09:31:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:48.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:31:48.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:31:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:31:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:31:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T09:31:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:31:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: stopped 2021-04-04T09:31:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:31:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:44 2021-04-04T09:31:48.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+stopped /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:48.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:31:48.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:48.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: stopped 2021-04-04T09:31:48.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:31:48.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:31:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:31:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:44 2021-04-04T09:31:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped 2021-04-04 09:31:44 2021-04-04T09:31:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:44 2021-04-04T09:31:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:31:48.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:44 2021-04-04T09:31:48.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:31:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+stopped 2021-04-04T09:31:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:31:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+stopped 2021-04-04T09:31:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:31:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2021-04-04T09:31:48.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped 2021-04-04T09:31:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test1.mirror_status 2021-04-04T09:31:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test1.mirror_status 2021-04-04T09:31:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test1 2021-04-04T09:31:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:49.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test1: 2021-04-04T09:31:49.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:31:49.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:31:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: stopped 2021-04-04T09:31:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2021-04-04T09:31:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:44 2021-04-04T09:31:49.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+stopped /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:31:49.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: stopped 2021-04-04T09:31:49.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:31:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:49.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:31:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:49.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:31:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:44 2021-04-04T09:31:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped 2021-04-04 09:31:44 2021-04-04T09:31:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:31:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:44 2021-04-04T09:31:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:49.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/last_update 2021-04-04T09:31:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:44 2021-04-04T09:31:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/state 2021-04-04T09:31:49.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+stopped 2021-04-04T09:31:49.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:31:49.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+stopped 2021-04-04T09:31:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:31:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:49.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror start mirror/test 2021-04-04T09:31:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:31:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror/test 2021-04-04T09:31:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:49.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:49.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:49.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:49.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror/test 2021-04-04T09:31:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:31:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror start mirror/test1 2021-04-04T09:31:49.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:49.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror start mirror/test 2021-04-04T09:31:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:49.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:49.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:49.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start mirror/test 2021-04-04T09:31:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-04-04T09:31:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-04-04T09:31:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:31:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2021-04-04T09:31:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:31:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:31:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:31:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + shift 2021-04-04T09:31:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:31:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:31:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:50.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:31:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:31:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:31:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:50.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:31:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:50.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:50.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:50.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:31:50.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:50.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:31:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:31:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:31:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:31:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:31:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:31:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2021-04-04T09:31:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2021-04-04T09:31:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:31:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:31:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:31:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:31:52.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T09:31:52.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:31:52.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T09:31:52.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:31:52.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:49 2021-04-04T09:31:52.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:31:52.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:52.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T09:31:52.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:52.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:52.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:52.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:31:52.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:49 2021-04-04T09:31:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2021-04-04 09:31:49 2021-04-04T09:31:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:49 2021-04-04T09:31:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:52.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:52.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:52.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:31:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:49 2021-04-04T09:31:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:31:52.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:31:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:31:52.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:52.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ all_admin_daemons cluster1 rbd mirror start mirror cluster2 2021-04-04T09:31:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror start mirror cluster2 2021-04-04T09:31:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:52.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:52.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:52.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:52.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:52.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror start mirror cluster2 2021-04-04T09:31:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror start mirror cluster2 2021-04-04T09:31:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:52.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:52.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:52.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:52.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror start mirror cluster2 2021-04-04T09:31:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror start mirror cluster2 2021-04-04T09:31:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:31:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:31:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:31:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:31:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:31:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:52.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:52.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:31:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:52.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror start mirror cluster2 2021-04-04T09:31:52.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror start mirror cluster2 2021-04-04T09:31:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:31:52.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:31:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:52.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:31:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:31:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:31:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:52.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:52.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:52.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:31:52.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror start mirror cluster2 2021-04-04T09:31:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2021-04-04T09:31:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2021-04-04T09:31:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:31:52.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test1 started 2021-04-04T09:31:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:31:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test1 2021-04-04T09:31:53.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:31:53.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:53.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:31:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test1 2021-04-04T09:31:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:53.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:53.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2021-04-04T09:31:53.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:31:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + echo "state": "Replaying" 2021-04-04T09:31:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i Replaying 2021-04-04T09:31:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:31:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:31:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:31:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2021-04-04T09:31:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying 2021-04-04T09:31:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:54.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:54.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test1.mirror_status 2021-04-04T09:31:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test1.mirror_status 2021-04-04T09:31:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test1 2021-04-04T09:31:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test1: 2021-04-04T09:31:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:31:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:31:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:31:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2021-04-04T09:31:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:52 2021-04-04T09:31:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:31:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:31:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:31:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:31:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:52 2021-04-04T09:31:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying 2021-04-04 09:31:52 2021-04-04T09:31:54.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:52 2021-04-04T09:31:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:54.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/last_update 2021-04-04T09:31:55.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:52 2021-04-04T09:31:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/state 2021-04-04T09:31:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:31:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:31:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror restart mirror/test 2021-04-04T09:31:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:31:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror/test 2021-04-04T09:31:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:55.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:55.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:55.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror/test 2021-04-04T09:31:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:31:55.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror restart mirror/test1 2021-04-04T09:31:55.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror/test 2021-04-04T09:31:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:55.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror/test 2021-04-04T09:31:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-04-04T09:31:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-04-04T09:31:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:31:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2021-04-04T09:31:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:31:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:31:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:31:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster=cluster1 2021-04-04T09:31:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:31:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:31:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:56.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:56.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:56.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:56.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:31:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:31:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:31:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:31:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:56.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:56.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:56.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:56.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:56.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:56.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:56.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:31:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:31:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:31:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:31:56.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:31:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:31:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:31:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2021-04-04T09:31:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2021-04-04T09:31:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:31:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:31:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:31:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:31:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + CEPH_ARGS= rbdtee --cluster /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status cluster1 2021-04-04T09:31:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: mirror image status mirror/test 2021-04-04T09:31:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:31:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T09:31:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:31:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T09:31:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:31:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:31:55 2021-04-04T09:31:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:31:57.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T09:31:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:31:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:31:57.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:31:57.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:31:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:31:55 2021-04-04T09:31:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2021-04-04 09:31:55 2021-04-04T09:31:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:57.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:31:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:31:55 2021-04-04T09:31:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:31:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:31:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:31:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:31:57.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:31:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:31:57.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:31:57.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:31:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:31:55 2021-04-04T09:31:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:31:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:31:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:31:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:31:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:31:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ all_admin_daemons cluster1 rbd mirror restart mirror cluster2 2021-04-04T09:31:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror cluster2 2021-04-04T09:31:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:57.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:57.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:57.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:57.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:57.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:57.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror cluster2 2021-04-04T09:31:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror cluster2 2021-04-04T09:31:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:57.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:57.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror cluster2 2021-04-04T09:31:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror restart mirror cluster2 2021-04-04T09:31:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:31:57.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:31:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:57.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:57.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:31:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:31:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:31:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:31:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:31:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:31:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:31:58.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror restart mirror cluster2 2021-04-04T09:31:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror restart mirror cluster2 2021-04-04T09:31:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:31:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:31:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:31:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:31:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:31:58.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:31:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:31:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:31:58.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:31:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:31:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:58.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:31:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror restart mirror cluster2 2021-04-04T09:31:58.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-04-04T09:31:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:58.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-04-04T09:31:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:58.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:31:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:31:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2021-04-04T09:31:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:31:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:31:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:31:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:31:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:31:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:31:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:31:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:31:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:31:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:31:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:31:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:31:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:31:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:31:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:31:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:31:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:31:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:31:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:31:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:59.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:31:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:31:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:31:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:31:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:31:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:31:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:31:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:31:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:31:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:31:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:31:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:31:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:31:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:31:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:31:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:31:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:31:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:31:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:31:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:31:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:31:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:31:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:31:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:31:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:31:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:31:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:31:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:31:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:31:59.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:31:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:31:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:31:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:31:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:31:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:31:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:31:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:31:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:31:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2021-04-04T09:31:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2021-04-04T09:31:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:31:59.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:31:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:31:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:31:59.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:31:59.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test1 started 2021-04-04T09:32:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:32:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:32:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test1 2021-04-04T09:32:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:32:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster=cluster1 2021-04-04T09:32:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ localgrep instance=cluster1 -i 2021-04-04T09:32:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state 2021-04-04T09:32:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test1 2021-04-04T09:32:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:00.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:00.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:00.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:00.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:00.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2021-04-04T09:32:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:00.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:32:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:32:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:32:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:32:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:32:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:32:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ all_admin_daemons cluster1 rbd mirror stop mirror cluster2 2021-04-04T09:32:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror stop mirror cluster2 2021-04-04T09:32:00.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:00.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:00.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror stop mirror cluster2 2021-04-04T09:32:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror stop mirror cluster2 2021-04-04T09:32:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:00.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:00.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:00.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:00.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:00.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:00.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:00.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:00.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:00.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror stop mirror cluster2 2021-04-04T09:32:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror stop mirror cluster2 2021-04-04T09:32:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:32:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:00.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:32:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:32:00.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:32:00.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:32:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:32:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror stop mirror cluster2 2021-04-04T09:32:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror stop mirror cluster2 2021-04-04T09:32:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:32:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:32:01.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:32:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:32:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:32:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:32:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:32:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:32:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:32:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:32:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:32:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:01.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:32:01.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:01.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror stop mirror cluster2 2021-04-04T09:32:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2021-04-04T09:32:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2021-04-04T09:32:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=stopped 2021-04-04T09:32:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:32:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test stopped 2021-04-04T09:32:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:02.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:32:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:02.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:32:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:32:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local+ cluster=cluster1 2021-04-04T09:32:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:32:02.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ localgrep loop_instance 2021-04-04T09:32:02.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: -i state 2021-04-04T09:32:02.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:32:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:02.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:02.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:02.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:02.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:02.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:02.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:02.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:02.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:32:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:32:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:32:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:02.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:02.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:02.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:02.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:02.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:32:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:32:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + grep -i Replaying 2021-04-04T09:32:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:echo "state": "Stopped" 2021-04-04T09:32:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = stopped 2021-04-04T09:32:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_stopped cluster1 mirror test1 2021-04-04T09:32:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:32:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test1 stopped 2021-04-04T09:32:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:32:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=stopped 2021-04-04T09:32:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:32:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test1 stopped 2021-04-04T09:32:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:32:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:32:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test1 2021-04-04T09:32:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:32:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + shift 2021-04-04T09:32:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:32:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:32:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test1 2021-04-04T09:32:03.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:03.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:03.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2021-04-04T09:32:03.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:32:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Stopped" 2021-04-04T09:32:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:32:03.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = stopped 2021-04-04T09:32:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2021-04-04T09:32:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:03.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:32:03.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped 2021-04-04T09:32:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:04.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:32:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:32:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:32:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test+ 2021-04-04T09:32:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:32:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T09:32:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:32:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: stopped 2021-04-04T09:32:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:32:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:00 2021-04-04T09:32:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+stopped /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:32:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: stopped 2021-04-04T09:32:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:32:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:32:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:32:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:32:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:32:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:32:00 2021-04-04T09:32:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped 2021-04-04 09:32:00 2021-04-04T09:32:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:32:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:32:00 2021-04-04T09:32:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:32:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:32:04.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:32:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:32:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:32:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:32:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:32:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:32:00 2021-04-04T09:32:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:32:04.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+stopped 2021-04-04T09:32:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:32:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+stopped 2021-04-04T09:32:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:32:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+stopped 2021-04-04T09:32:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:32:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:32:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:04.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:04.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+stopped 2021-04-04T09:32:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:32:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:32:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test1.mirror_status 2021-04-04T09:32:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test1.mirror_status 2021-04-04T09:32:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:32:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test1 2021-04-04T09:32:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:32:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test1: 2021-04-04T09:32:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:32:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:32:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: stopped 2021-04-04T09:32:05.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2021-04-04T09:32:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:00 2021-04-04T09:32:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+stopped /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:32:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:32:05.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:32:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: stopped 2021-04-04T09:32:05.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:32:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + echo up+stopped 2021-04-04T09:32:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep ^up+ 2021-04-04T09:32:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:32:05.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:32:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:32:05.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:32:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:32:00 2021-04-04T09:32:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+stopped 2021-04-04 09:32:00 2021-04-04T09:32:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:32:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:32:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:32:00 2021-04-04T09:32:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:32:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:32:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:32:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:32:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:32:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:32:05.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/last_update 2021-04-04T09:32:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:32:00 2021-04-04T09:32:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/state 2021-04-04T09:32:05.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+stopped 2021-04-04T09:32:05.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:32:05.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+stopped 2021-04-04T09:32:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:32:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ all_admin_daemons cluster1 rbd mirror restart mirror cluster2 2021-04-04T09:32:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror cluster2 2021-04-04T09:32:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror cluster2 2021-04-04T09:32:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror cluster2 2021-04-04T09:32:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:06.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:06.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:06.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:06.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror cluster2 2021-04-04T09:32:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror restart mirror cluster2 2021-04-04T09:32:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:32:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:32:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:06.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:32:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:32:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:32:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:06.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:06.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:32:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror restart mirror cluster2 2021-04-04T09:32:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror restart mirror cluster2 2021-04-04T09:32:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:32:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:06.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:32:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:32:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:06.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:32:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:32:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:32:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:32:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:32:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:32:06.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:32:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:32:06.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:06.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:32:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror restart mirror cluster2 2021-04-04T09:32:06.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-04-04T09:32:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:06.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:06.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-04-04T09:32:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:06.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:32:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:32:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2021-04-04T09:32:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:32:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:32:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:32:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:32:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:32:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:07.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:32:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:07.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:07.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:07.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:07.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:07.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:07.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:07.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:07.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:32:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:32:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:07.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:32:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:07.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:07.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:07.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:07.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:07.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:07.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:07.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:07.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:07.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:32:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:32:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:32:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:32:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:32:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:32:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:32:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test1 2021-04-04T09:32:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:32:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test1 started 2021-04-04T09:32:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:32:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:32:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:32:07.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test1 started 2021-04-04T09:32:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:08.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:32:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:32:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test1 2021-04-04T09:32:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:32:08.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state+ 2021-04-04T09:32:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:shift 2021-04-04T09:32:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:32:08.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:08.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:08.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test1 2021-04-04T09:32:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:08.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test1 2021-04-04T09:32:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:32:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:32:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:32:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:32:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:32:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:32:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2021-04-04T09:32:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:32:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:08.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2021-04-04T09:32:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:32:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:32:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:32:09.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:09.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:32:09.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:32:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T09:32:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:32:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T09:32:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:32:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:06 2021-04-04T09:32:09.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:09.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:32:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T09:32:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:32:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:32:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:32:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:32:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:09.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:32:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:32:06 2021-04-04T09:32:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2021-04-04 09:32:06 2021-04-04T09:32:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:32:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:32:06 2021-04-04T09:32:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:32:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:32:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:09.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:32:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:32:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:32:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:32:10.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:32:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:32:06 2021-04-04T09:32:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:32:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:32:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:32:10.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:32:10.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:32:10.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test1 up+replaying 2021-04-04T09:32:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:32:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:32:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test1 up+replaying 2021-04-04T09:32:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:32:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:32:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test1.mirror_status 2021-04-04T09:32:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test1.mirror_status 2021-04-04T09:32:11.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:32:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test1 2021-04-04T09:32:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:32:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test1: 2021-04-04T09:32:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:32:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:32:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:32:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2021-04-04T09:32:11.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:06 2021-04-04T09:32:11.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:32:11.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:32:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:32:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:32:11.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:32:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + echo up+replaying 2021-04-04T09:32:11.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep ^up+ 2021-04-04T09:32:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:32:11.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:32:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:32:11.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test1.mirror_status 2021-04-04T09:32:11.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:32:06 2021-04-04T09:32:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test1 up+replaying 2021-04-04 09:32:06 2021-04-04T09:32:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:11.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test1 2021-04-04T09:32:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:32:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:32:06 2021-04-04T09:32:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:32:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:32:11.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:11.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:32:11.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:32:11.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:32:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:32:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/last_update 2021-04-04T09:32:11.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:32:06 2021-04-04T09:32:11.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test1']/state 2021-04-04T09:32:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:32:11.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:32:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:32:11.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:32:11.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:11.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 2021-04-04T09:32:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool= 2021-04-04T09:32:11.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image= 2021-04-04T09:32:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:32:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:32:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush 2021-04-04T09:32:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:32:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:32:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush 2021-04-04T09:32:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:11.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:11.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:11.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:11.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:11.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:11.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush 2021-04-04T09:32:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ all_admin_daemons cluster1 rbd mirror status 2021-04-04T09:32:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status 2021-04-04T09:32:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status 2021-04-04T09:32:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2021-04-04T09:32:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool_replayers": [ 2021-04-04T09:32:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "peer": "uuid: ba5e7ea1-e930-4550-b570-945e322d5ab4 cluster: cluster2 client: client.mirror", 2021-04-04T09:32:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool": "mirror", 2021-04-04T09:32:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4317", 2021-04-04T09:32:11.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "stopped (manual)", 2021-04-04T09:32:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader_instance_id": "4317", 2021-04-04T09:32:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader": true, 2021-04-04T09:32:11.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instances": [ 2021-04-04T09:32:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4317", 2021-04-04T09:32:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4320", 2021-04-04T09:32:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4327", 2021-04-04T09:32:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4330" 2021-04-04T09:32:11.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ], 2021-04-04T09:32:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.0.15736.cluster1.94301502485824.asok", 2021-04-04T09:32:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.15736.cluster2.94301514495296.asok", 2021-04-04T09:32:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "sync_throttler": { 2021-04-04T09:32:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 5, 2021-04-04T09:32:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:32:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:32:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "deletion_throttler": { 2021-04-04T09:32:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 1, 2021-04-04T09:32:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:32:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:32:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [ 2021-04-04T09:32:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "mirror/test1", 2021-04-04T09:32:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:32:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:32:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ], 2021-04-04T09:32:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter": { 2021-04-04T09:32:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter_status": { 2021-04-04T09:32:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "delete_images_queue": [], 2021-04-04T09:32:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "failed_deletes_queue": [] 2021-04-04T09:32:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:32:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "namespaces": [ 2021-04-04T09:32:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "ns1", 2021-04-04T09:32:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [], 2021-04-04T09:32:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter": { 2021-04-04T09:32:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter_status": { 2021-04-04T09:32:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "delete_images_queue": [], 2021-04-04T09:32:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "failed_deletes_queue": [] 2021-04-04T09:32:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:32:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:32:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "ns2", 2021-04-04T09:32:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [], 2021-04-04T09:32:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter": { 2021-04-04T09:32:11.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter_status": { 2021-04-04T09:32:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "delete_images_queue": [], 2021-04-04T09:32:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "failed_deletes_queue": [] 2021-04-04T09:32:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:32:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:32:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:32:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ] 2021-04-04T09:32:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "peer": "uuid: 062cbc91-4713-4ded-b744-6fda24ecf93a cluster: cluster2 client: client.mirror", 2021-04-04T09:32:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool": "mirror_parent", 2021-04-04T09:32:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4333", 2021-04-04T09:32:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "stopped (manual)", 2021-04-04T09:32:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader_instance_id": "4333", 2021-04-04T09:32:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader": true, 2021-04-04T09:32:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instances": [ 2021-04-04T09:32:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4333", 2021-04-04T09:32:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4335", 2021-04-04T09:32:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4338", 2021-04-04T09:32:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4341" 2021-04-04T09:32:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ], 2021-04-04T09:32:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.0.15736.cluster1.94301527651648.asok", 2021-04-04T09:32:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.15736.cluster2.94301540013376.asok", 2021-04-04T09:32:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "sync_throttler": { 2021-04-04T09:32:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 5, 2021-04-04T09:32:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:32:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:32:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "deletion_throttler": { 2021-04-04T09:32:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 1, 2021-04-04T09:32:11.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:32:11.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:32:11.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [], 2021-04-04T09:32:11.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter": { 2021-04-04T09:32:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter_status": { 2021-04-04T09:32:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "delete_images_queue": [], 2021-04-04T09:32:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "failed_deletes_queue": [] 2021-04-04T09:32:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:32:11.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "namespaces": [] 2021-04-04T09:32:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:32:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ] 2021-04-04T09:32:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2021-04-04T09:32:11.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status 2021-04-04T09:32:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:11.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:11.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:11.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:11.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:11.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:11.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:11.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:11.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status 2021-04-04T09:32:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2021-04-04T09:32:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool_replayers": [ 2021-04-04T09:32:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "peer": "uuid: ba5e7ea1-e930-4550-b570-945e322d5ab4 cluster: cluster2 client: client.mirror", 2021-04-04T09:32:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool": "mirror", 2021-04-04T09:32:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4320", 2021-04-04T09:32:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "stopped (manual)", 2021-04-04T09:32:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader_instance_id": "4317", 2021-04-04T09:32:11.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader": false, 2021-04-04T09:32:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.1.15764.cluster1.94557984707904.asok", 2021-04-04T09:32:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.15764.cluster2.94557996897600.asok", 2021-04-04T09:32:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "sync_throttler": { 2021-04-04T09:32:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 5, 2021-04-04T09:32:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:32:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:32:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "deletion_throttler": { 2021-04-04T09:32:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 1, 2021-04-04T09:32:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:32:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:32:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [ 2021-04-04T09:32:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "mirror/test", 2021-04-04T09:32:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:32:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:32:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ], 2021-04-04T09:32:11.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "namespaces": [ 2021-04-04T09:32:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "ns1", 2021-04-04T09:32:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [] 2021-04-04T09:32:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "ns2", 2021-04-04T09:32:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [] 2021-04-04T09:32:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:32:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ] 2021-04-04T09:32:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "peer": "uuid: 062cbc91-4713-4ded-b744-6fda24ecf93a cluster: cluster2 client: client.mirror", 2021-04-04T09:32:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool": "mirror_parent", 2021-04-04T09:32:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4335", 2021-04-04T09:32:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "stopped (manual)", 2021-04-04T09:32:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader_instance_id": "4333", 2021-04-04T09:32:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader": false, 2021-04-04T09:32:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.1.15764.cluster1.94558009849152.asok", 2021-04-04T09:32:11.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.15764.cluster2.94558022137152.asok", 2021-04-04T09:32:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "sync_throttler": { 2021-04-04T09:32:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 5, 2021-04-04T09:32:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:32:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:32:11.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "deletion_throttler": { 2021-04-04T09:32:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 1, 2021-04-04T09:32:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:32:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:32:11.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [], 2021-04-04T09:32:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "namespaces": [] 2021-04-04T09:32:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:32:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ] 2021-04-04T09:32:11.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2021-04-04T09:32:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status 2021-04-04T09:32:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:32:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:11.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:32:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:32:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:32:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:32:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:11.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:32:11.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:11.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status 2021-04-04T09:32:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2021-04-04T09:32:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool_replayers": [ 2021-04-04T09:32:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "peer": "uuid: ba5e7ea1-e930-4550-b570-945e322d5ab4 cluster: cluster2 client: client.mirror", 2021-04-04T09:32:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool": "mirror", 2021-04-04T09:32:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4327", 2021-04-04T09:32:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "stopped (manual)", 2021-04-04T09:32:11.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader_instance_id": "4317", 2021-04-04T09:32:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader": false, 2021-04-04T09:32:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.2.15792.cluster1.94644719826240.asok", 2021-04-04T09:32:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.15792.cluster2.94644731819328.asok", 2021-04-04T09:32:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "sync_throttler": { 2021-04-04T09:32:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 5, 2021-04-04T09:32:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:32:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:32:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "deletion_throttler": { 2021-04-04T09:32:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 1, 2021-04-04T09:32:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:32:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:32:11.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [], 2021-04-04T09:32:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "namespaces": [ 2021-04-04T09:32:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "ns1", 2021-04-04T09:32:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [] 2021-04-04T09:32:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "ns2", 2021-04-04T09:32:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [] 2021-04-04T09:32:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:32:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ] 2021-04-04T09:32:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "peer": "uuid: 062cbc91-4713-4ded-b744-6fda24ecf93a cluster: cluster2 client: client.mirror", 2021-04-04T09:32:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool": "mirror_parent", 2021-04-04T09:32:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4338", 2021-04-04T09:32:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "stopped (manual)", 2021-04-04T09:32:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader_instance_id": "4333", 2021-04-04T09:32:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader": false, 2021-04-04T09:32:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.2.15792.cluster1.94644745033024.asok", 2021-04-04T09:32:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.15792.cluster2.94644757042496.asok", 2021-04-04T09:32:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "sync_throttler": { 2021-04-04T09:32:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 5, 2021-04-04T09:32:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:32:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:32:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "deletion_throttler": { 2021-04-04T09:32:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 1, 2021-04-04T09:32:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:32:11.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:32:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [], 2021-04-04T09:32:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "namespaces": [] 2021-04-04T09:32:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:32:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ] 2021-04-04T09:32:11.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2021-04-04T09:32:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status 2021-04-04T09:32:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:32:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:11.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:32:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:32:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:11.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:32:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:11.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:11.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:32:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:32:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:32:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:32:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:32:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:32:11.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:32:11.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:11.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:32:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status 2021-04-04T09:32:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2021-04-04T09:32:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool_replayers": [ 2021-04-04T09:32:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "peer": "uuid: ba5e7ea1-e930-4550-b570-945e322d5ab4 cluster: cluster2 client: client.mirror", 2021-04-04T09:32:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool": "mirror", 2021-04-04T09:32:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4330", 2021-04-04T09:32:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "stopped (manual)", 2021-04-04T09:32:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader_instance_id": "4317", 2021-04-04T09:32:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader": false, 2021-04-04T09:32:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.3.15852.cluster1.94044427875648.asok", 2021-04-04T09:32:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.15852.cluster2.94044440499520.asok", 2021-04-04T09:32:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "sync_throttler": { 2021-04-04T09:32:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 5, 2021-04-04T09:32:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:32:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:32:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "deletion_throttler": { 2021-04-04T09:32:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 1, 2021-04-04T09:32:11.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:32:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:32:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [], 2021-04-04T09:32:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "namespaces": [ 2021-04-04T09:32:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "ns1", 2021-04-04T09:32:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [] 2021-04-04T09:32:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "ns2", 2021-04-04T09:32:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [] 2021-04-04T09:32:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:32:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ] 2021-04-04T09:32:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:32:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "peer": "uuid: 062cbc91-4713-4ded-b744-6fda24ecf93a cluster: cluster2 client: client.mirror", 2021-04-04T09:32:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool": "mirror_parent", 2021-04-04T09:32:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4341", 2021-04-04T09:32:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "stopped (manual)", 2021-04-04T09:32:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader_instance_id": "4333", 2021-04-04T09:32:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader": false, 2021-04-04T09:32:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.3.15852.cluster1.94044453516608.asok", 2021-04-04T09:32:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.15852.cluster2.94044465706304.asok", 2021-04-04T09:32:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "sync_throttler": { 2021-04-04T09:32:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 5, 2021-04-04T09:32:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:32:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:32:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "deletion_throttler": { 2021-04-04T09:32:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 1, 2021-04-04T09:32:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:32:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:32:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:32:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [], 2021-04-04T09:32:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "namespaces": [] 2021-04-04T09:32:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:32:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ] 2021-04-04T09:32:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2021-04-04T09:32:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog TEST: test image rename 2021-04-04T09:32:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:32:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:32:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:32:11 TEST: test image rename 2021-04-04T09:32:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:32:11 TEST: test image rename 2021-04-04T09:32:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ new_name=test_RENAMED 2021-04-04T09:32:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_image cluster2 mirror test test_RENAMED 2021-04-04T09:32:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_name=test_RENAMED 2021-04-04T09:32:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 rename mirror/test mirror/test_RENAMED 2021-04-04T09:32:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test_RENAMED 2021-04-04T09:32:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_RENAMED 2021-04-04T09:32:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test_RENAMED started 2021-04-04T09:32:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_RENAMED 2021-04-04T09:32:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:32:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:32:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:12.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test_RENAMED started 2021-04-04T09:32:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_RENAMED 2021-04-04T09:32:12.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:32:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:32:12.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shiftgrep 2021-04-04T09:32:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: -i state 2021-04-04T09:32:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:12.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:32:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:12.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:12.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:12.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:12.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2021-04-04T09:32:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:13.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:13.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:13.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:13.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:13.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2021-04-04T09:32:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:32:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:32:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:13.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:13.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:13.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:32:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:32:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:32:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:13.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:13.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:32:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:13.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:13.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:13.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2021-04-04T09:32:13.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:13.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:32:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:32:13.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:32:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:32:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:13.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:32:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:32:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:32:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:32:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:32:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:32:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:32:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:13.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:32:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2021-04-04T09:32:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:14.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:14.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:14.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:14.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2021-04-04T09:32:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:14.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:14.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:14.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:32:14.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:32:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:32:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:32:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:32:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:32:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying 2021-04-04T09:32:14.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:14.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_RENAMED 2021-04-04T09:32:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:32:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying 2021-04-04T09:32:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_RENAMED 2021-04-04T09:32:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:32:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test_RENAMED.mirror_status 2021-04-04T09:32:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test_RENAMED.mirror_status 2021-04-04T09:32:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_RENAMED.mirror_status 2021-04-04T09:32:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test_RENAMED 2021-04-04T09:32:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_RENAMED.mirror_status 2021-04-04T09:32:15.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test_RENAMED: 2021-04-04T09:32:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T09:32:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:32:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T09:32:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:32:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:06 2021-04-04T09:32:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_RENAMED.mirror_status 2021-04-04T09:32:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:32:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_RENAMED.mirror_status 2021-04-04T09:32:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T09:32:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:32:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:32:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:32:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:32:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_RENAMED.mirror_status 2021-04-04T09:32:15.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:32:15.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_RENAMED.mirror_status 2021-04-04T09:32:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:32:06 2021-04-04T09:32:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_RENAMED up+replaying 2021-04-04 09:32:06 2021-04-04T09:32:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_RENAMED 2021-04-04T09:32:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:32:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:32:06 2021-04-04T09:32:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:32:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:32:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:32:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:32:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:32:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:32:15.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_RENAMED']/last_update 2021-04-04T09:32:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:32:06 2021-04-04T09:32:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_RENAMED']/state 2021-04-04T09:32:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:32:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:32:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:32:15.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:32:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:32:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:15.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:32:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:15.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:15.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:15.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:15.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:15.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:15.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:15.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:15.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:15.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:16.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2021-04-04T09:32:16.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:16.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:16.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:16.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:16.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:16.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:16.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2021-04-04T09:32:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "mirror/test_RENAMED", 2021-04-04T09:32:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:32:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2021-04-04T09:32:16.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror restart mirror/test_RENAMED 2021-04-04T09:32:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:32:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:32:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror restart mirror/test_RENAMED 2021-04-04T09:32:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:16.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:16.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror restart mirror/test_RENAMED 2021-04-04T09:32:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2021-04-04T09:32:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror restart : restart rbd mirror 2021-04-04T09:32:16.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror restart mirror/test_RENAMED 2021-04-04T09:32:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:16.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:16.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:16.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:16.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:16.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:16.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:16.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror restart mirror/test_RENAMED 2021-04-04T09:32:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test_RENAMED 2021-04-04T09:32:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_RENAMED 2021-04-04T09:32:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test_RENAMED started 2021-04-04T09:32:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_RENAMED 2021-04-04T09:32:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:32:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:32:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test_RENAMED started 2021-04-04T09:32:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_RENAMED 2021-04-04T09:32:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:32:17.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:17.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:32:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1+ 2021-04-04T09:32:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:17.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:32:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:17.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:17.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:17.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:17.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:17.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:17.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:17.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:17.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_RENAMED'] 2021-04-04T09:32:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:17.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:17.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:17.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:17.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:17.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:17.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:17.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:17.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:17.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:17.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:32:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:32:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:32:17.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:32:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:32:17.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:32:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying 2021-04-04T09:32:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_RENAMED 2021-04-04T09:32:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:32:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test_RENAMED up+replaying 2021-04-04T09:32:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:18.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_RENAMED 2021-04-04T09:32:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:32:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test_RENAMED.mirror_status 2021-04-04T09:32:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test_RENAMED.mirror_status 2021-04-04T09:32:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_RENAMED.mirror_status 2021-04-04T09:32:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test_RENAMED+ 2021-04-04T09:32:18.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_RENAMED.mirror_status 2021-04-04T09:32:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test_RENAMED: 2021-04-04T09:32:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T09:32:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:32:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6}} 2021-04-04T09:32:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:32:18.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:16 2021-04-04T09:32:18.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_RENAMED.mirror_status 2021-04-04T09:32:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:32:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_RENAMED.mirror_status 2021-04-04T09:32:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":6}} 2021-04-04T09:32:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:32:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:32:18.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:32:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:32:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_RENAMED.mirror_status 2021-04-04T09:32:18.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:32:18.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_RENAMED.mirror_status 2021-04-04T09:32:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:32:16 2021-04-04T09:32:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_RENAMED up+replaying 2021-04-04 09:32:16 2021-04-04T09:32:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_RENAMED 2021-04-04T09:32:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:32:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:32:16 2021-04-04T09:32:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:32:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:32:18.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:32:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:32:18.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:32:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:32:18.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_RENAMED']/last_update 2021-04-04T09:32:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:32:16 2021-04-04T09:32:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_RENAMED']/state 2021-04-04T09:32:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:32:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:32:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:32:18.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:32:18.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_image cluster2 mirror test_RENAMED test 2021-04-04T09:32:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:18.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_RENAMED 2021-04-04T09:32:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_name=test 2021-04-04T09:32:18.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 rename mirror/test_RENAMED mirror/test 2021-04-04T09:32:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-04-04T09:32:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-04-04T09:32:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:18.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:32:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:32:18.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2021-04-04T09:32:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:19.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:32:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:19.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:32:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1+ 2021-04-04T09:32:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i+ statelocal 2021-04-04T09:32:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: cluster=cluster1 2021-04-04T09:32:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:32:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:32:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:19.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:19.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:19.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:19.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:19.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:19.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:19.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:19.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:19.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:32:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:32:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:32:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:20.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:20.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:20.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:20.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:20.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:20.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:32:20.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test_RENAMED 2021-04-04T09:32:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-04-04T09:32:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:32:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:32:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:32:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:32:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:32:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:20.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:32:20.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:20.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:20.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-04-04T09:32:20.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:32:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:20.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2021-04-04T09:32:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:32:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:32:20.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:32:20.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:20.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:20.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:32:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:32:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:32:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:32:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:32:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:32:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:32:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:32:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:20.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:32:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:20.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2021-04-04T09:32:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:32:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:21.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:32:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:21.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:32:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:32:21.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:32:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:21.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:21.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:21.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:21.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:21.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:21.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:21.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:32:21.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:32:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:32:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:32:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:32:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:32:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:32:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog TEST: test trash move restore 2021-04-04T09:32:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + date +%F %T 2021-04-04T09:32:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:32:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:32:21 TEST: test trash move restore 2021-04-04T09:32:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:32:21 TEST: test trash move restore 2021-04-04T09:32:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster2 mirror test 2021-04-04T09:32:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:21.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 info mirror/test 2021-04-04T09:32:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:32:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id=1099ce7c2fd0 2021-04-04T09:32:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ trash_move cluster2 mirror test 2021-04-04T09:32:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:21.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 trash move mirror/test 2021-04-04T09:32:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror test deleted 2021-04-04T09:32:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=deleted 2021-04-04T09:32:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:32:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:32:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:32:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test 2021-04-04T09:32:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test 2021-04-04T09:32:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:32:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id=10a6c20c33e5 2021-04-04T09:32:21.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:32:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test deleted 10a6c20c33e5 2021-04-04T09:32:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:21.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=deleted 2021-04-04T09:32:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id=10a6c20c33e5 2021-04-04T09:32:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:32:21.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:32:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test 2021-04-04T09:32:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test 2021-04-04T09:32:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:32:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test: (2) No such file or directory 2021-04-04T09:32:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:32:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:32:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test deleted = deleted 2021-04-04T09:32:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ trash_restore cluster2 mirror 1099ce7c2fd0 2021-04-04T09:32:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id=1099ce7c2fd0 2021-04-04T09:32:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 trash restore mirror/1099ce7c2fd0 2021-04-04T09:32:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-04-04T09:32:22.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-04-04T09:32:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:22.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:32:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:32:22.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2021-04-04T09:32:23.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:32:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:32:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + localgrep cluster_instance=cluster1 -i 2021-04-04T09:32:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state 2021-04-04T09:32:23.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:32:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:23.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:23.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:32:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:23.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:23.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:23.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:23.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:23.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:23.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:32:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:32:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:32:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:32:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:32:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-04-04T09:32:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:32:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:23.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:32:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:23.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:32:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:32:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:32:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:32:23.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:23.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-04-04T09:32:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:32:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:23.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2021-04-04T09:32:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:32:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:32:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:32:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:32:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:23.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:23.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:32:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:32:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:32:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:32:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:32:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:32:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:32:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:32:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2021-04-04T09:32:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:32:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:23.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:24.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:32:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:32:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:32:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:32:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:24.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:24.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:24.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:24.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:32:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:32:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:24.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-04-04T09:32:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:32:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:32:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:32:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:32:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:32:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:24.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:32:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-04-04T09:32:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:32:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2021-04-04T09:32:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:32:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:32:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:32:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:32:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:32:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:32:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:32:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:32:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:32:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:32:24.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:32:24.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:24.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:32:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:32:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2021-04-04T09:32:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:32:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:25.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:32:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:32:27.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:27.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:27.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:32:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:32:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:32:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:27.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:27.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:27.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:32:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:32:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-04-04T09:32:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:32:27.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:32:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:27.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:27.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:27.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:27.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:27.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:27.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:32:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:32:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:32:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:32:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:27.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:27.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-04-04T09:32:27.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:32:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:32:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:32:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:32:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:32:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:32:27.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog TEST: failover and failback 2021-04-04T09:32:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:32:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:32:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:32:27 TEST: failover and failback 2021-04-04T09:32:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:32:27 TEST: failover and failback 2021-04-04T09:32:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirrors cluster2 2021-04-04T09:32:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:27.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster2:0 2021-04-04T09:32:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:32:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:32:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:32:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:27.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:32:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:32:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:27.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:32:27.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:32:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster2:1 2021-04-04T09:32:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:32:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:27.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:32:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:32:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:32:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:32:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:32:27.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd-mirror --cluster cluster2 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:32:27.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster2:2 2021-04-04T09:32:27.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:32:27.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:32:27.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:32:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:32:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:32:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:32:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd-mirror --cluster cluster2 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:32:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster2:3 2021-04-04T09:32:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:32:27.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:27.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:32:27.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:32:27.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:27.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:27.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:27.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:27.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:32:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:32:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:32:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:32:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:32:27.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd-mirror --cluster cluster2 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:32:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ demote_image cluster2 mirror test 2021-04-04T09:32:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:27.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:27.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:27.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2021-04-04T09:32:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Image demoted to non-primary 2021-04-04T09:32:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2021-04-04T09:32:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2021-04-04T09:32:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=stopped 2021-04-04T09:32:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:32:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:28.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test stopped 2021-04-04T09:32:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:32:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:32:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local+ cluster_instance=cluster1 2021-04-04T09:32:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state+ 2021-04-04T09:32:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:local cluster=cluster1 2021-04-04T09:32:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:32:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:28.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:32:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:28.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:32:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:32:28.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:32:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:28.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:28.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:28.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:28.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:32:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:32:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-04-04T09:32:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:32:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:32:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:32:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:32:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:32:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:32:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:28.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-04-04T09:32:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:32:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + echo "state": "Stopped" 2021-04-04T09:32:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i Replaying 2021-04-04T09:32:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = stopped 2021-04-04T09:32:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2021-04-04T09:32:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:32:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown 2021-04-04T09:32:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:32:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:32:30.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:32:30.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:30.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster+ cluster1 mirror image status mirror/test 2021-04-04T09:32:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:32:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:32:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:32:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: stopped 2021-04-04T09:32:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2021-04-04T09:32:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:28 2021-04-04T09:32:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:32:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:32:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:32:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:32:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:29 2021-04-04T09:32:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+unknown /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 1 2021-04-04T09:32:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:32:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown 2021-04-04T09:32:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:32.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:32:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:32:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:32:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:32:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:32:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:32:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:32:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:32:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2021-04-04T09:32:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:30 2021-04-04T09:32:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:32:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:32:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:32:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:32:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:29 2021-04-04T09:32:32.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+unknown /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+unknown 2021-04-04T09:32:32.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: remote image is non-primary 2021-04-04T09:32:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:32:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:32:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:32:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:32:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2021-04-04T09:32:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:32:30 2021-04-04T09:32:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown 2021-04-04 09:32:30 2021-04-04T09:32:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:32:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:32:30 2021-04-04T09:32:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:32:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:32:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:32:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:32:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:32:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:32:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:32:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:32:30 2021-04-04T09:32:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:32:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+unknown 2021-04-04T09:32:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:32:32.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+unknown 2021-04-04T09:32:32.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:32:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2021-04-04T09:32:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:32:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown 2021-04-04T09:32:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:32:33.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.mirror_status 2021-04-04T09:32:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.mirror_status 2021-04-04T09:32:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:33.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror image status mirror/test 2021-04-04T09:32:33.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:33.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:32:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:32:33.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:32:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:32:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:32:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:32 2021-04-04T09:32:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:32:33.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster1 2021-04-04T09:32:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:32:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:32:33.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:30 2021-04-04T09:32:33.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+unknown /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+unknown 2021-04-04T09:32:33.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: remote image is non-primary 2021-04-04T09:32:33.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:32:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + echo up+unknown 2021-04-04T09:32:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep ^up+ 2021-04-04T09:32:33.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:32:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:32:33.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:32:32 2021-04-04T09:32:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown 2021-04-04 09:32:32 2021-04-04T09:32:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:33.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:32:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:32:32 2021-04-04T09:32:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror.mirror_status 2021-04-04T09:32:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror.mirror_status 2021-04-04T09:32:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:33.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror.mirror_status 2021-04-04T09:32:33.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2021-04-04T09:32:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:32:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:32:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:32:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:32:32 2021-04-04T09:32:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:32:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+unknown 2021-04-04T09:32:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:32:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+unknown 2021-04-04T09:32:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:32:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ promote_image cluster2 mirror test 2021-04-04T09:32:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:33.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local force= 2021-04-04T09:32:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2021-04-04T09:32:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Image promoted to primary 2021-04-04T09:32:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-04-04T09:32:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-04-04T09:32:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:32:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:32:33.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2021-04-04T09:32:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:34.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:32:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:34.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:34.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:32:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:32:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:32:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:32:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:32:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:34.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:34.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:34.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:32:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:32:34.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:32:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:34.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:34.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:34.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:34.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:34.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:34.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:34.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:32:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:32:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:34.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-04-04T09:32:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:32:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:32:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:34.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:34.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:32:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:32:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:32:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:34.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:32:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:34.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-04-04T09:32:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:32:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + grep -i Replaying 2021-04-04T09:32:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:echo "state": "Stopped" 2021-04-04T09:32:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = stopped 2021-04-04T09:32:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:32:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2021-04-04T09:32:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:32:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:32:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:37.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:37.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:32:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:37.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:37.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:37.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:32:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:32:37.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:37.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:32:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:32:37.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-04-04T09:32:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:32:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:32:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:32:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:32:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:32:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:32:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:37.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-04-04T09:32:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:32:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:32:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:32:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:32:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:32:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:32:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test 100 2021-04-04T09:32:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:32:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:32:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:32:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:32:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:32:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:32:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:32:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2272.73 bytes/sec: 8.9 MiB/s 2021-04-04T09:32:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2021-04-04T09:32:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:32:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:32:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2021-04-04T09:32:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:32:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:32:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:32:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:32:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:32:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test 2021-04-04T09:32:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:32:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:32:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test 2021-04-04T09:32:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2021-04-04T09:32:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:32:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:32:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2021-04-04T09:32:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:37.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2021-04-04T09:32:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush mirror/test1 2021-04-04T09:32:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2021-04-04T09:32:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:37.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:37.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2021-04-04T09:32:37.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:32:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:32:37.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2021-04-04T09:32:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:32:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:32:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:37.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:32:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:32:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:32:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:32:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2021-04-04T09:32:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test 2021-04-04T09:32:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test 2021-04-04T09:32:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:32:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:32:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:32:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:32:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:32:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:32:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:32:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=12, entry_tid=101], [object_number=0, tag_tid=12, entry_tid=100], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:32:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=3, tag_tid=8, entry_tid=3], [object_number=2, tag_tid=8, entry_tid=2], [object_number=1, tag_tid=8, entry_tid=1]]], state=connected] 2021-04-04T09:32:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=12, entry_tid=101 2021-04-04T09:32:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test 2021-04-04T09:32:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test ..* 2021-04-04T09:32:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:32:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:32:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:32:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:32:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:32:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:32:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:32:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=12, entry_tid=101], [object_number=0, tag_tid=12, entry_tid=100], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:32:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=3, tag_tid=8, entry_tid=3], [object_number=2, tag_tid=8, entry_tid=2], [object_number=1, tag_tid=8, entry_tid=1]]], state=connected] 2021-04-04T09:32:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=0, tag_tid=11, entry_tid=0 2021-04-04T09:32:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=12, entry_tid=101 -a object_number=1, tag_tid=12, entry_tid=101 = object_number=0, tag_tid=11, entry_tid=0 2021-04-04T09:32:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=0, tag_tid=11, entry_tid=0 != 2021-04-04T09:32:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:32:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=0, tag_tid=11, entry_tid=0 = 2021-04-04T09:32:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_mirror_pos=object_number=0, tag_tid=11, entry_tid=0 2021-04-04T09:32:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=12, entry_tid=101 2021-04-04T09:32:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + cutgrep -d= -f -Eo 2 2021-04-04T09:32:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: tag_tid=[0-9]* 2021-04-04T09:32:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_tag=12 2021-04-04T09:32:38.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=0, tag_tid=11, entry_tid=0 2021-04-04T09:32:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + cut -d=grep -f -Eo 2 2021-04-04T09:32:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: tag_tid=[0-9]* 2021-04-04T09:32:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_tag=11 2021-04-04T09:32:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=12, entry_tid=101 2021-04-04T09:32:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + grep -Eo entry_tid=[0-9]* 2021-04-04T09:32:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:cut -d= -f 2 2021-04-04T09:32:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_entry=101 2021-04-04T09:32:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=0, tag_tid=11, entry_tid=0 2021-04-04T09:32:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo entry_tid=[0-9]* 2021-04-04T09:32:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:32:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_entry=0 2021-04-04T09:32:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 12 = 11 -a 101 -le 0 2021-04-04T09:32:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:32:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:32:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test 2021-04-04T09:32:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:38.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:32:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:32:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test 2021-04-04T09:32:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2021-04-04T09:32:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:32:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:32:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2021-04-04T09:32:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:38.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:38.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2021-04-04T09:32:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush mirror/test1 2021-04-04T09:32:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2021-04-04T09:32:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:38.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:38.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:38.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:38.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:38.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:38.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2021-04-04T09:32:38.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:32:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:32:38.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2021-04-04T09:32:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:32:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:38.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:32:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:38.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:38.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:38.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:38.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:32:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:32:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:32:38.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:38.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:38.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:38.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:38.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:38.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:38.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:32:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:38.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2021-04-04T09:32:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test 2021-04-04T09:32:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test 2021-04-04T09:32:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:32:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:32:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:32:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image+ mirror/test 2021-04-04T09:32:38.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:32:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:32:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:32:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=12, entry_tid=101], [object_number=0, tag_tid=12, entry_tid=100], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:32:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=3, tag_tid=8, entry_tid=3], [object_number=2, tag_tid=8, entry_tid=2], [object_number=1, tag_tid=8, entry_tid=1]]], state=connected] 2021-04-04T09:32:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=12, entry_tid=101 2021-04-04T09:32:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test 2021-04-04T09:32:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:38.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test ..* 2021-04-04T09:32:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:38.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:32:38.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:32:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:32:38.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:32:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:32:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:32:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:32:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=12, entry_tid=101], [object_number=0, tag_tid=12, entry_tid=100], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:32:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0], [object_number=3, tag_tid=8, entry_tid=3], [object_number=2, tag_tid=8, entry_tid=2], [object_number=1, tag_tid=8, entry_tid=1]]], state=connected] 2021-04-04T09:32:38.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=0, tag_tid=11, entry_tid=0 2021-04-04T09:32:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=12, entry_tid=101 -a object_number=1, tag_tid=12, entry_tid=101 = object_number=0, tag_tid=11, entry_tid=0 2021-04-04T09:32:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=0, tag_tid=11, entry_tid=0 != object_number=0, tag_tid=11, entry_tid=0 2021-04-04T09:32:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2021-04-04T09:32:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test 2021-04-04T09:32:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:32:39.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:32:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test 2021-04-04T09:32:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2021-04-04T09:32:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:32:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:39.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:32:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2021-04-04T09:32:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:39.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:39.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2021-04-04T09:32:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush mirror/test1 2021-04-04T09:32:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2021-04-04T09:32:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:39.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:39.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:39.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2021-04-04T09:32:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:32:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:32:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2021-04-04T09:32:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:32:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:32:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:32:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:32:39.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:32:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:39.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:32:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2021-04-04T09:32:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test 2021-04-04T09:32:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:39.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test 2021-04-04T09:32:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:39.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:32:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:32:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:32:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:32:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:39.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:32:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:32:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:32:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=12, entry_tid=101], [object_number=0, tag_tid=12, entry_tid=100], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:32:39.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=12, entry_tid=101], [object_number=0, tag_tid=12, entry_tid=100], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:32:39.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=12, entry_tid=101 2021-04-04T09:32:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test 2021-04-04T09:32:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test ..* 2021-04-04T09:32:39.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:32:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:32:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:32:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:32:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:32:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:32:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:32:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=12, entry_tid=101], [object_number=0, tag_tid=12, entry_tid=100], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:32:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=12, entry_tid=101], [object_number=0, tag_tid=12, entry_tid=100], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:32:39.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:32:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=1, tag_tid=12, entry_tid=101 2021-04-04T09:32:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=12, entry_tid=101 -a object_number=1, tag_tid=12, entry_tid=101 = object_number=1, tag_tid=12, entry_tid=101 2021-04-04T09:32:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2021-04-04T09:32:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:39.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:32:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:39.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped 2021-04-04T09:32:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:32:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:40.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:40.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.mirror_status 2021-04-04T09:32:40.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.mirror_status 2021-04-04T09:32:40.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror image status mirror/test 2021-04-04T09:32:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:32:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:32:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:32:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:32:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:32:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:37 2021-04-04T09:32:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:32:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster1 2021-04-04T09:32:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:32:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":10},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}} 2021-04-04T09:32:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:35 2021-04-04T09:32:40.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+stopped /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:32:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:40.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: local image is primary 2021-04-04T09:32:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:32:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:32:40.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:32:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:32:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:32:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:32:37 2021-04-04T09:32:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped 2021-04-04 09:32:37 2021-04-04T09:32:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:32:40.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:32:37 2021-04-04T09:32:40.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror.mirror_status 2021-04-04T09:32:40.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror.mirror_status 2021-04-04T09:32:40.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror.mirror_status 2021-04-04T09:32:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2021-04-04T09:32:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:32:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:32:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:32:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:32:37 2021-04-04T09:32:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:32:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+stopped 2021-04-04T09:32:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:32:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+stopped 2021-04-04T09:32:41.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:32:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2021-04-04T09:32:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:32:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:32:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2021-04-04T09:32:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:42.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:32:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:32:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:32:42.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:32:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:32:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:32:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:32:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:32:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":10},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}} 2021-04-04T09:32:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2021-04-04T09:32:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:35 2021-04-04T09:32:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:32:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:32:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:32:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:32:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:37 2021-04-04T09:32:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:32:42.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .*primary_position /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:42.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":10},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}} 2021-04-04T09:32:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:32:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:32:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:32:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:32:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2021-04-04T09:32:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:32:35 2021-04-04T09:32:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2021-04-04 09:32:35 2021-04-04T09:32:42.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:32:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:32:35 2021-04-04T09:32:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:32:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:32:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:32:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:32:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:32:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:32:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:32:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:32:35 2021-04-04T09:32:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:32:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:32:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:32:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:32:42.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:32:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_images mirror test 2021-04-04T09:32:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2021-04-04T09:32:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.export 2021-04-04T09:32:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.export 2021-04-04T09:32:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export 2021-04-04T09:32:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.export 2021-04-04T09:32:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.export 2021-04-04T09:32:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:32:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:32:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export 2021-04-04T09:32:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:32:42.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:32:42.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:32:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:32:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:32:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ demote_image cluster2 mirror test 2021-04-04T09:32:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2021-04-04T09:32:42.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Image demoted to non-primary 2021-04-04T09:32:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2021-04-04T09:32:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2021-04-04T09:32:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=stopped 2021-04-04T09:32:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:32:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test stopped 2021-04-04T09:32:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:32:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:32:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:32:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster=cluster1 2021-04-04T09:32:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ greplocal instance=cluster1 -i 2021-04-04T09:32:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state 2021-04-04T09:32:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:32:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:32:43.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:32:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:32:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:43.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:43.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:32:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:32:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:43.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:32:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:32:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:43.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:32:43.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:43.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:43.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:32:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:32:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:32:43.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:43.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:32:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:32:43.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:32:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:43.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:32:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:43.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:32:43.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:32:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:32:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:32:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:43.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:32:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:43.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:32:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:32:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:32:44.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:32:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:32:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-04-04T09:32:44.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:32:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:44.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:44.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:32:44.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:44.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:44.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:32:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:32:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:32:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:32:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:32:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:32:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:32:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:32:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:32:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:32:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:32:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:32:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:32:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-04-04T09:32:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:32:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Stopped" 2021-04-04T09:32:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:32:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = stopped 2021-04-04T09:32:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2021-04-04T09:32:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:32:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:44.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown 2021-04-04T09:32:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:45.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:32:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:45.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:32:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:32:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:45.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:32:45.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:32:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:32:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:32:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: stopped 2021-04-04T09:32:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2021-04-04T09:32:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:44 2021-04-04T09:32:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:32:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:32:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:32:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:32:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:42 2021-04-04T09:32:45.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+unknown /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:45.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 1 2021-04-04T09:32:45.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:32:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown 2021-04-04T09:32:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:32:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:32:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:32:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:32:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:32:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:32:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:32:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:32:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2021-04-04T09:32:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:45 2021-04-04T09:32:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:32:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:32:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:32:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:32:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:42 2021-04-04T09:32:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+unknown /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+unknown 2021-04-04T09:32:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: remote image is non-primary 2021-04-04T09:32:47.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:32:47.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:32:47.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:32:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:32:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2021-04-04T09:32:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:32:45 2021-04-04T09:32:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown 2021-04-04 09:32:45 2021-04-04T09:32:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:32:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:32:45 2021-04-04T09:32:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:32:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:32:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:47.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:32:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:32:47.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:32:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:32:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:32:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:32:45 2021-04-04T09:32:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:32:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+unknown 2021-04-04T09:32:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:32:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+unknown 2021-04-04T09:32:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:32:47.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:47.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:47.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2021-04-04T09:32:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:32:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown 2021-04-04T09:32:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:32:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.mirror_status 2021-04-04T09:32:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.mirror_status 2021-04-04T09:32:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:48.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror image status mirror/test 2021-04-04T09:32:48.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:48.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:32:48.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:32:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:32:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:32:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:32:48.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:47 2021-04-04T09:32:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:32:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster1 2021-04-04T09:32:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:32:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:32:48.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:45 2021-04-04T09:32:48.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+unknown /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+unknown 2021-04-04T09:32:48.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:48.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: remote image is non-primary 2021-04-04T09:32:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:32:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:32:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:32:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:32:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:32:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:32:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:32:47 2021-04-04T09:32:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown 2021-04-04 09:32:47 2021-04-04T09:32:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:32:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:32:47 2021-04-04T09:32:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror.mirror_status 2021-04-04T09:32:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror.mirror_status 2021-04-04T09:32:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror.mirror_status 2021-04-04T09:32:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2021-04-04T09:32:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:32:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:32:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:32:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:32:47 2021-04-04T09:32:48.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:32:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+unknown 2021-04-04T09:32:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + grep up+unknown 2021-04-04T09:32:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:echo up+unknown 2021-04-04T09:32:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:32:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ promote_image cluster1 mirror test 2021-04-04T09:32:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local force= 2021-04-04T09:32:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2021-04-04T09:32:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Image promoted to primary 2021-04-04T09:32:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster2 mirror test 2021-04-04T09:32:48.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2021-04-04T09:32:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:32:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:32:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster2 mirror test started 2021-04-04T09:32:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:32:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror status mirror/test 2021-04-04T09:32:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster2 2021-04-04T09:32:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + shift 2021-04-04T09:32:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:32:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:49.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster2 2021-04-04T09:32:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:49.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:32:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status mirror/test 2021-04-04T09:32:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:32:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:32:49.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:32:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:32:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:32:49.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:32:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:32:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:32:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:32:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:49.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:32:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:32:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:32:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:32:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:32:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:32:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:32:49.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:32:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:32:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status mirror/test 2021-04-04T09:32:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:32:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:32:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:32:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:32:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:32:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:32:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:32:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:32:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:32:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:32:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:32:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:32:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:32:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:32:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:32:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:32:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:32:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:32:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Stopped" 2021-04-04T09:32:49.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:32:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = stopped 2021-04-04T09:32:49.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:32:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster2 mirror test started 2021-04-04T09:32:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:51.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:32:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror status mirror/test 2021-04-04T09:32:51.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster2 2021-04-04T09:32:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local instance=cluster2 2021-04-04T09:32:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:51.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i+ state 2021-04-04T09:32:51.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:sleep 0 2021-04-04T09:32:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:32:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status mirror/test 2021-04-04T09:32:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:32:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:32:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:32:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:51.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:32:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:32:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:51.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:32:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:32:51.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:32:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:32:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:51.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:51.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:32:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:32:51.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:32:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:51.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:32:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:32:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:32:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:32:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:32:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:52.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:32:52.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status mirror/test 2021-04-04T09:32:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:32:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:32:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:32:52.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:32:52.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:32:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:52.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:32:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:32:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:32:52.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:32:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:32:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:32:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:32:52.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:52.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:32:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:32:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:32:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:32:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:32:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:32:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Stopped" 2021-04-04T09:32:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:32:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = stopped 2021-04-04T09:32:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 4 2021-04-04T09:32:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster2 mirror test started 2021-04-04T09:32:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:32:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror status mirror/test 2021-04-04T09:32:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster2 2021-04-04T09:32:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local instance=cluster2 2021-04-04T09:32:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state+ 2021-04-04T09:32:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:sleep 0 2021-04-04T09:32:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:32:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status mirror/test 2021-04-04T09:32:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:32:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:32:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:32:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:32:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:32:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:56.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:32:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:32:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:32:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:32:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:32:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:32:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:32:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:32:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:32:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:32:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:32:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:32:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:32:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status mirror/test 2021-04-04T09:32:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:32:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:56.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:32:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:32:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:32:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:32:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:32:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:32:56.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:32:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:32:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:56.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:32:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:32:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:32:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:56.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:32:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:32:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:32:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:32:56.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:32:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:32:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:32:56.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:32:56.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:32:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:32:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:56.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ demote_image cluster1 mirror test 2021-04-04T09:32:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2021-04-04T09:32:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Image demoted to non-primary 2021-04-04T09:32:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2021-04-04T09:32:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2021-04-04T09:32:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=stopped 2021-04-04T09:32:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:32:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster2 mirror test stopped 2021-04-04T09:32:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:32:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:32:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:32:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror status mirror/test 2021-04-04T09:32:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster2 2021-04-04T09:32:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + shift 2021-04-04T09:32:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i+ state 2021-04-04T09:32:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:local cluster=cluster2 2021-04-04T09:32:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster2 2021-04-04T09:32:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:32:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:32:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:32:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:32:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status mirror/test 2021-04-04T09:32:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:32:57.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:32:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:32:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:32:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:32:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:57.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:32:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:32:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:32:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:32:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:57.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:32:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:32:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:32:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:32:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:32:57.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:32:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:32:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:32:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:32:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:32:57.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:32:57.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:32:57.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:32:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status mirror/test 2021-04-04T09:32:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:32:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:32:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:32:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:32:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:32:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:57.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:32:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:32:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:32:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:32:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:32:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:32:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:32:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:32:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:32:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:32:57.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:32:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:32:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:32:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:32:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:32:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:32:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:32:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:32:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:32:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:32:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:32:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:32:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + echo "state": "Stopped" 2021-04-04T09:32:57.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i Replaying 2021-04-04T09:32:57.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = stopped 2021-04-04T09:32:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:32:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2021-04-04T09:32:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:32:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:32:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown 2021-04-04T09:32:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:32:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:32:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:32:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:32:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:32:58.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:32:58.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:32:58.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:32:58.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:32:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:32:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:32:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:32:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:32:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:32:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2021-04-04T09:32:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:55 2021-04-04T09:32:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:32:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:32:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:32:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: stopped 2021-04-04T09:32:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:57 2021-04-04T09:32:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+unknown /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:32:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 1 2021-04-04T09:32:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:33:00.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown 2021-04-04T09:33:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:00.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:33:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:33:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:33:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:33:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:33:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:33:01.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:33:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2021-04-04T09:33:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:00 2021-04-04T09:33:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:01.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:33:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: stopped 2021-04-04T09:33:01.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:57 2021-04-04T09:33:01.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+unknown /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+unknown 2021-04-04T09:33:01.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: remote image is non-primary 2021-04-04T09:33:01.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:01.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:33:01.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:33:01.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:33:01.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:01.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2021-04-04T09:33:01.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:33:00 2021-04-04T09:33:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown 2021-04-04 09:33:00 2021-04-04T09:33:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:01.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:33:00 2021-04-04T09:33:01.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:33:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:33:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:01.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:33:01.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:33:01.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:33:01.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:33:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:33:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:33:00 2021-04-04T09:33:01.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:33:01.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+unknown 2021-04-04T09:33:01.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:33:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+unknown 2021-04-04T09:33:01.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:33:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2021-04-04T09:33:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:01.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown 2021-04-04T09:33:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.mirror_status 2021-04-04T09:33:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.mirror_status 2021-04-04T09:33:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2+ mirror image status mirror/test 2021-04-04T09:33:02.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:33:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:33:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: stopped 2021-04-04T09:33:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:33:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:32:57 2021-04-04T09:33:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster1 2021-04-04T09:33:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:33:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:33:02.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:00 2021-04-04T09:33:02.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+unknown /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 1 2021-04-04T09:33:02.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:33:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown 2021-04-04T09:33:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:04.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.mirror_status 2021-04-04T09:33:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.mirror_status 2021-04-04T09:33:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror image status mirror/test 2021-04-04T09:33:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:33:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:33:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:33:04.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:33:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:33:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:02 2021-04-04T09:33:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:04.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster1 2021-04-04T09:33:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:33:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:33:04.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:00 2021-04-04T09:33:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+unknown /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+unknown 2021-04-04T09:33:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:04.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: remote image is non-primary 2021-04-04T09:33:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + grep ^up+ 2021-04-04T09:33:04.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:echo up+unknown 2021-04-04T09:33:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:33:04.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:33:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:33:02 2021-04-04T09:33:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown 2021-04-04 09:33:02 2021-04-04T09:33:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:04.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:04.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:33:02 2021-04-04T09:33:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror.mirror_status 2021-04-04T09:33:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror.mirror_status 2021-04-04T09:33:04.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror.mirror_status 2021-04-04T09:33:04.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2021-04-04T09:33:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:33:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:33:04.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:33:04.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:33:02 2021-04-04T09:33:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:33:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+unknown 2021-04-04T09:33:04.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:33:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+unknown 2021-04-04T09:33:04.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:33:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ promote_image cluster2 mirror test 2021-04-04T09:33:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:04.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local force= 2021-04-04T09:33:04.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2021-04-04T09:33:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Image promoted to primary 2021-04-04T09:33:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-04-04T09:33:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-04-04T09:33:04.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:33:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:33:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:05.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2021-04-04T09:33:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:05.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:33:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:33:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:33:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + grep -i state 2021-04-04T09:33:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:33:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:33:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:33:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:33:05.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:05.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:33:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:33:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:33:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:05.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:33:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:33:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:33:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:05.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:33:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:33:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:33:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:33:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:33:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:33:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:33:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:33:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:33:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:33:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:05.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:05.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:33:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:33:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:33:05.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-04-04T09:33:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:33:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:33:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:05.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:33:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:33:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:33:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:05.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:05.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:05.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:33:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:05.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-04-04T09:33:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:33:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:33:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:33:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:33:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:33:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:33:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2021-04-04T09:33:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:33:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:33:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2021-04-04T09:33:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:33:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:33:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:33:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:33:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:33:06.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test 2021-04-04T09:33:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:06.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:33:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:33:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test 2021-04-04T09:33:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2021-04-04T09:33:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:33:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:33:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:33:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2021-04-04T09:33:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:33:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:33:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:33:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:33:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:33:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:33:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:06.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2021-04-04T09:33:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:33:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush mirror/test1 2021-04-04T09:33:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2021-04-04T09:33:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:33:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:33:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:33:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:33:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:33:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:06.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:33:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2021-04-04T09:33:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:33:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:33:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2021-04-04T09:33:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:33:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:33:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:33:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:33:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:33:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:33:06.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:06.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2021-04-04T09:33:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test 2021-04-04T09:33:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:06.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:06.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test 2021-04-04T09:33:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:33:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:33:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:33:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:33:06.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:33:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:33:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:33:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=12, entry_tid=101], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:33:06.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=12, entry_tid=101], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:33:06.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=0, tag_tid=17, entry_tid=0 2021-04-04T09:33:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test 2021-04-04T09:33:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test ..* 2021-04-04T09:33:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:33:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:33:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:33:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:33:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:33:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:33:06.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:33:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=12, entry_tid=101], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:33:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=0, tag_tid=17, entry_tid=0], [object_number=1, tag_tid=12, entry_tid=101], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:33:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=0, tag_tid=17, entry_tid=0 2021-04-04T09:33:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=0, tag_tid=17, entry_tid=0 -a object_number=0, tag_tid=17, entry_tid=0 = object_number=0, tag_tid=17, entry_tid=0 2021-04-04T09:33:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2021-04-04T09:33:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:33:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:33:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:06.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2021-04-04T09:33:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:33:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:33:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:33:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:33:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:33:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:33:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:33:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:33:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":14},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}} 2021-04-04T09:33:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2021-04-04T09:33:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:05 2021-04-04T09:33:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:33:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:33:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:07 2021-04-04T09:33:07.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:07.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:33:07.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .*primary_position /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:07.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":14},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}} 2021-04-04T09:33:07.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:33:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:33:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:33:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:07.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2021-04-04T09:33:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:33:05 2021-04-04T09:33:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2021-04-04 09:33:05 2021-04-04T09:33:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:33:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:33:05 2021-04-04T09:33:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:33:07.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:33:07.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:08.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:33:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:33:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:33:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:33:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:33:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:33:05 2021-04-04T09:33:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:33:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:33:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + grep up+replaying 2021-04-04T09:33:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:echo up+replaying 2021-04-04T09:33:08.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:33:08.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2021-04-04T09:33:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:08.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:09.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped 2021-04-04T09:33:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:09.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.mirror_status 2021-04-04T09:33:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.mirror_status 2021-04-04T09:33:09.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror image status mirror/test 2021-04-04T09:33:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:09.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:33:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:33:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:09.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:33:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:33:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:07 2021-04-04T09:33:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster1 2021-04-04T09:33:09.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:33:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":14},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":17}} 2021-04-04T09:33:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:05 2021-04-04T09:33:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+stopped /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:09.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:33:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:09.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: local image is primary 2021-04-04T09:33:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:33:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:33:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:33:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:33:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:33:07 2021-04-04T09:33:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped 2021-04-04 09:33:07 2021-04-04T09:33:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:33:07 2021-04-04T09:33:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror.mirror_status 2021-04-04T09:33:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror.mirror_status 2021-04-04T09:33:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror.mirror_status 2021-04-04T09:33:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2021-04-04T09:33:09.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:33:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:33:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:33:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:33:07 2021-04-04T09:33:09.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:33:09.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+stopped 2021-04-04T09:33:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:33:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+stopped 2021-04-04T09:33:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:33:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_images mirror test 2021-04-04T09:33:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2021-04-04T09:33:09.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.export 2021-04-04T09:33:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.export 2021-04-04T09:33:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export 2021-04-04T09:33:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.export 2021-04-04T09:33:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.export 2021-04-04T09:33:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:33:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:33:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export 2021-04-04T09:33:09.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:33:09.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:33:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:33:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:33:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:33:09.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ demote_image cluster2 mirror test 2021-04-04T09:33:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:09.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image demote mirror/test 2021-04-04T09:33:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Image demoted to non-primary 2021-04-04T09:33:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_stopped cluster1 mirror test 2021-04-04T09:33:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test stopped 2021-04-04T09:33:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=stopped 2021-04-04T09:33:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:33:09.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test stopped 2021-04-04T09:33:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:33:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:33:10.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:33:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:33:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:33:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i+ statelocal 2021-04-04T09:33:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: cluster=cluster1 2021-04-04T09:33:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:33:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:33:10.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:33:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:33:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:10.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:33:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:10.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:10.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:33:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:33:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:33:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:10.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:33:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:33:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:33:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:33:11.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:33:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:33:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:33:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:11.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:11.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:11.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:11.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:33:11.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:33:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:33:11.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:11.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:11.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:33:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:11.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:33:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:33:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:11.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-04-04T09:33:11.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:33:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:33:11.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:11.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:33:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:33:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:33:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:33:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-04-04T09:33:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:11.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:33:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Stopped" 2021-04-04T09:33:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:33:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = stopped 2021-04-04T09:33:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2021-04-04T09:33:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:11.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:11.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:11.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:11.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:11.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown 2021-04-04T09:33:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:33:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:33:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:33:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:33:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:33:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: stopped 2021-04-04T09:33:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2021-04-04T09:33:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:10 2021-04-04T09:33:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:33:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:33:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:07 2021-04-04T09:33:12.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+unknown /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:12.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 1 2021-04-04T09:33:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:33:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown 2021-04-04T09:33:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:33:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:33:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:33:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:33:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:33:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: stopped 2021-04-04T09:33:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2021-04-04T09:33:14.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:10 2021-04-04T09:33:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:33:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:33:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:33:14.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:12 2021-04-04T09:33:14.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+unknown /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 1 2021-04-04T09:33:14.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 4 2021-04-04T09:33:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown 2021-04-04T09:33:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:33:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:33:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:33:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:33:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:33:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:33:18.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:33:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2021-04-04T09:33:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:15 2021-04-04T09:33:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:18.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:33:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:33:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:33:18.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:17 2021-04-04T09:33:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+unknown /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:18.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+unknown 2021-04-04T09:33:18.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: remote image is non-primary 2021-04-04T09:33:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:33:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:33:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:33:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2021-04-04T09:33:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:33:15 2021-04-04T09:33:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown 2021-04-04 09:33:15 2021-04-04T09:33:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:33:15 2021-04-04T09:33:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:33:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:33:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:33:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:33:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:33:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:33:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:33:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:33:15 2021-04-04T09:33:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:33:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+unknown 2021-04-04T09:33:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:33:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+unknown 2021-04-04T09:33:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:33:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2021-04-04T09:33:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown 2021-04-04T09:33:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.mirror_status 2021-04-04T09:33:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.mirror_status 2021-04-04T09:33:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror image status mirror/test 2021-04-04T09:33:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:33:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:33:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:33:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:33:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:33:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:17 2021-04-04T09:33:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster1 2021-04-04T09:33:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:33:19.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:33:19.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:15 2021-04-04T09:33:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+unknown /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+unknown 2021-04-04T09:33:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: remote image is non-primary 2021-04-04T09:33:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:33:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:33:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:33:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:33:19.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:33:17 2021-04-04T09:33:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown 2021-04-04 09:33:17 2021-04-04T09:33:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:33:17 2021-04-04T09:33:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror.mirror_status 2021-04-04T09:33:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror.mirror_status 2021-04-04T09:33:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror.mirror_status 2021-04-04T09:33:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2021-04-04T09:33:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:33:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:33:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:33:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:33:17 2021-04-04T09:33:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:33:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+unknown 2021-04-04T09:33:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:33:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+unknown 2021-04-04T09:33:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:33:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ promote_image cluster1 mirror test 2021-04-04T09:33:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local force= 2021-04-04T09:33:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test 2021-04-04T09:33:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Image promoted to primary 2021-04-04T09:33:19.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster2 mirror test 2021-04-04T09:33:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:19.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster2 mirror test started 2021-04-04T09:33:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:19.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:33:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:33:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster2 mirror test started 2021-04-04T09:33:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:20.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:33:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:33:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:33:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror status mirror/test 2021-04-04T09:33:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster2 2021-04-04T09:33:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + shift 2021-04-04T09:33:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i+ state 2021-04-04T09:33:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:local cluster=cluster2 2021-04-04T09:33:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster2 2021-04-04T09:33:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:33:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status mirror/test 2021-04-04T09:33:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:33:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:33:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:20.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:33:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:33:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:20.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:33:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:33:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:20.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:33:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:33:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:33:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status mirror/test 2021-04-04T09:33:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:33:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:33:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:33:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:33:21.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:21.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:21.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:33:21.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:21.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:21.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:21.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:21.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:21.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:33:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:33:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:33:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Stopped" 2021-04-04T09:33:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:33:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = stopped 2021-04-04T09:33:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:33:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster2 mirror test started 2021-04-04T09:33:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:23.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:33:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:33:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:33:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror status mirror/test 2021-04-04T09:33:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster2 2021-04-04T09:33:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + shift 2021-04-04T09:33:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state+ 2021-04-04T09:33:23.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:local cluster=cluster2 2021-04-04T09:33:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster2 2021-04-04T09:33:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:23.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:23.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:33:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status mirror/test 2021-04-04T09:33:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:33:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:33:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:33:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:33:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:33:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:33:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:33:23.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:33:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:33:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status mirror/test 2021-04-04T09:33:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:33:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:33:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:23.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:23.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:33:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:33:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:23.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:33:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:23.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:33:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:33:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:33:23.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:33:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:33:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:33:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:33:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:33:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster1 mirror test 100 2021-04-04T09:33:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:33:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:33:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:33:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:33:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:33:23.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:33:23.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:33:23.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2777.78 bytes/sec: 11 MiB/s 2021-04-04T09:33:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster2 cluster1 mirror test 2021-04-04T09:33:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2 2021-04-04T09:33:23.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:33:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster2 cluster1 mirror test 2021-04-04T09:33:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2 2021-04-04T09:33:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:33:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:33:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:33:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:33:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster2 mirror test 2021-04-04T09:33:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:33:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:33:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test 2021-04-04T09:33:23.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2021-04-04T09:33:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster2 2021-04-04T09:33:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster2 2021-04-04T09:33:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:33:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2021-04-04T09:33:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:33:23.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:33:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:23.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:33:23.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:33:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:33:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:23.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:23.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:23.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:33:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:23.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2021-04-04T09:33:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:33:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush mirror/test1 2021-04-04T09:33:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror flush mirror/test 2021-04-04T09:33:23.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:33:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:33:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:23.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:23.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:33:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:33:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:23.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:33:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:23.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:23.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:33:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror flush mirror/test 2021-04-04T09:33:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster1 mirror test 2021-04-04T09:33:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:24.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster1 mirror test 2021-04-04T09:33:24.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:33:24.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:33:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:33:24.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:24.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 journal status --image mirror/test 2021-04-04T09:33:24.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:33:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:33:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:33:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=12, entry_tid=101], [object_number=0, tag_tid=12, entry_tid=100], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:33:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0]]], state=connected] 2021-04-04T09:33:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=12, entry_tid=101 2021-04-04T09:33:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster1 mirror test 2021-04-04T09:33:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster1 mirror test ..* 2021-04-04T09:33:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:24.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:33:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:33:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:33:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 journal status --image mirror/test+ 2021-04-04T09:33:24.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:33:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:33:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:33:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=12, entry_tid=101], [object_number=0, tag_tid=12, entry_tid=100], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:33:24.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0]]], state=connected] 2021-04-04T09:33:24.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos= 2021-04-04T09:33:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=12, entry_tid=101 -a object_number=1, tag_tid=12, entry_tid=101 = 2021-04-04T09:33:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test != 2021-04-04T09:33:24.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2021-04-04T09:33:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster2 mirror test 2021-04-04T09:33:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:33:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:33:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test 2021-04-04T09:33:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2021-04-04T09:33:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster2 2021-04-04T09:33:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster2 2021-04-04T09:33:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:33:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2021-04-04T09:33:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:33:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:33:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:33:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:33:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:33:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:24.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:33:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2021-04-04T09:33:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:33:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush mirror/test1 2021-04-04T09:33:24.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror flush mirror/test 2021-04-04T09:33:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:33:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:33:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:33:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:33:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:33:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:33:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:24.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror flush mirror/test 2021-04-04T09:33:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster1 mirror test 2021-04-04T09:33:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster1 mirror test 2021-04-04T09:33:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:33:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:33:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:33:24.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 journal status --image mirror/test 2021-04-04T09:33:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:24.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:33:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:33:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:33:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=12, entry_tid=101], [object_number=0, tag_tid=12, entry_tid=100], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:33:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0]]], state=connected] 2021-04-04T09:33:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=12, entry_tid=101 2021-04-04T09:33:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster1 mirror test 2021-04-04T09:33:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster1 mirror test ..* 2021-04-04T09:33:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:33:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:33:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:33:24.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:24.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 journal status --image mirror/test 2021-04-04T09:33:24.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:33:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:33:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:33:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=12, entry_tid=101], [object_number=0, tag_tid=12, entry_tid=100], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:33:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[[object_number=0, tag_tid=11, entry_tid=0]]], state=connected] 2021-04-04T09:33:24.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos= 2021-04-04T09:33:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=12, entry_tid=101 -a object_number=1, tag_tid=12, entry_tid=101 = 2021-04-04T09:33:24.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test != 2021-04-04T09:33:24.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2021-04-04T09:33:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster2 mirror test 2021-04-04T09:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:25.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test 2021-04-04T09:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror flush mirror/test 2021-04-04T09:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster2 2021-04-04T09:33:25.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster2 2021-04-04T09:33:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:25.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:33:25.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror flush mirror/test 2021-04-04T09:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:33:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:25.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:33:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:33:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:33:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:25.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:33:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror flush mirror/test 2021-04-04T09:33:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:33:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush mirror/test1 2021-04-04T09:33:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror flush mirror/test 2021-04-04T09:33:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:33:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:33:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:25.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:33:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:33:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:33:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:25.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:33:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror flush mirror/test 2021-04-04T09:33:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster1 mirror test 2021-04-04T09:33:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster1 mirror test 2021-04-04T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:33:26.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:33:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 journal status --image mirror/test 2021-04-04T09:33:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:33:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:33:26.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:33:26.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=12, entry_tid=101], [object_number=0, tag_tid=12, entry_tid=100], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:33:26.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[[object_number=1, tag_tid=12, entry_tid=101], [object_number=0, tag_tid=12, entry_tid=100], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:33:26.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:26.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=12, entry_tid=101 2021-04-04T09:33:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster1 mirror test 2021-04-04T09:33:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster1 mirror test ..* 2021-04-04T09:33:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:26.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:33:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:33:26.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:33:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 journal status --image mirror/test 2021-04-04T09:33:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:33:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:33:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:33:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=12, entry_tid=101], [object_number=0, tag_tid=12, entry_tid=100], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:33:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[[object_number=1, tag_tid=12, entry_tid=101], [object_number=0, tag_tid=12, entry_tid=100], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:33:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=1, tag_tid=12, entry_tid=101 2021-04-04T09:33:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=12, entry_tid=101 -a object_number=1, tag_tid=12, entry_tid=101 = object_number=1, tag_tid=12, entry_tid=101 2021-04-04T09:33:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+stopped 2021-04-04T09:33:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+stopped 2021-04-04T09:33:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:33:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:33:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:33:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:33:27.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:33:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:33:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2021-04-04T09:33:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:25 2021-04-04T09:33:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:33:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:33:27.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":8},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":11}} 2021-04-04T09:33:27.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:22 2021-04-04T09:33:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+stopped /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:27.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:33:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: local image is primary 2021-04-04T09:33:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:33:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:33:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:33:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2021-04-04T09:33:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:33:25 2021-04-04T09:33:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+stopped 2021-04-04 09:33:25 2021-04-04T09:33:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:27.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:33:25 2021-04-04T09:33:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:33:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:33:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:27.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:33:27.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:33:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:33:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:33:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:33:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:33:25 2021-04-04T09:33:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:33:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+stopped 2021-04-04T09:33:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + echogrep up+stopped 2021-04-04T09:33:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: up+stopped 2021-04-04T09:33:27.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:33:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:27.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+replaying primary_position 2021-04-04T09:33:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:27.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:33:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:33:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster2 mirror test up+replaying primary_position 2021-04-04T09:33:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:33:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:33:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.mirror_status 2021-04-04T09:33:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.mirror_status 2021-04-04T09:33:28.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror image status mirror/test 2021-04-04T09:33:28.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:33:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:33:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:33:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":103518.0,"entries_behind_primary":0,"entries_per_second":25.75,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":12},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":12}} 2021-04-04T09:33:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:33:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:27 2021-04-04T09:33:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster1 2021-04-04T09:33:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:33:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:25 2021-04-04T09:33:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:33:28.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .*primary_position /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":103518.0,"entries_behind_primary":0,"entries_per_second":25.75,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":12},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":12}} 2021-04-04T09:33:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:33:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:33:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:33:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:33:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:33:27 2021-04-04T09:33:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+replaying 2021-04-04 09:33:27 2021-04-04T09:33:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:33:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:33:27 2021-04-04T09:33:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror.mirror_status 2021-04-04T09:33:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror.mirror_status 2021-04-04T09:33:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror.mirror_status 2021-04-04T09:33:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2021-04-04T09:33:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:33:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:33:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:33:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:33:27 2021-04-04T09:33:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:33:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:33:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + echo up+replaying 2021-04-04T09:33:28.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep up+replaying 2021-04-04T09:33:28.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:33:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:28.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_images mirror test 2021-04-04T09:33:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2021-04-04T09:33:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.export 2021-04-04T09:33:28.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.export 2021-04-04T09:33:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export 2021-04-04T09:33:28.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.export 2021-04-04T09:33:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.export 2021-04-04T09:33:28.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:33:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:33:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export 2021-04-04T09:33:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:33:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:33:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:33:28.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:33:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:33:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ demote_image cluster1 mirror test 2021-04-04T09:33:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:29.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/test 2021-04-04T09:33:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Image demoted to non-primary 2021-04-04T09:33:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_stopped cluster2 mirror test 2021-04-04T09:33:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:29.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster2 mirror test stopped 2021-04-04T09:33:29.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:29.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:29.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:29.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=stopped 2021-04-04T09:33:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:33:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster2 mirror test stopped 2021-04-04T09:33:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:30.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:33:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:33:30.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:33:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror status mirror/test 2021-04-04T09:33:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster2 2021-04-04T09:33:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grepshift 2021-04-04T09:33:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: -i state 2021-04-04T09:33:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster2 2021-04-04T09:33:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:30.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:33:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status mirror/test 2021-04-04T09:33:30.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:33:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:33:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:30.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:30.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:30.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:30.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:30.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:30.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:33:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:33:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:33:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:33:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:30.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:33:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:33:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:33:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status mirror/test 2021-04-04T09:33:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:33:30.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:33:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:33:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:33:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:33:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:33:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:30.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:33:30.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:33:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:33:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:33:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:33:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:33:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = started 2021-04-04T09:33:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:33:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster2 mirror test stopped 2021-04-04T09:33:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:33:32.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:33:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:33:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror status mirror/test 2021-04-04T09:33:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + grep -i state 2021-04-04T09:33:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:local cluster_instance=cluster2 2021-04-04T09:33:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster2 2021-04-04T09:33:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:33:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status mirror/test 2021-04-04T09:33:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:33:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:33:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:33:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:33:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:33:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:32.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:33:32.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:33:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:33:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:33:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status mirror/test 2021-04-04T09:33:32.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:33:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:33:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:32.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:32.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:33:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:33:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:32.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:33:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:32.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:33:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:33:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopping" 2021-04-04T09:33:32.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Stopping" 2021-04-04T09:33:32.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:33:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = stopped 2021-04-04T09:33:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+unknown 2021-04-04T09:33:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+unknown 2021-04-04T09:33:33.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:33.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:33.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:33:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:33:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:33:33.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:33:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:33:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:33:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:33:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2021-04-04T09:33:33.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:30 2021-04-04T09:33:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:33:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:33:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:33:33.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:32 2021-04-04T09:33:33.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+unknown /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+unknown 2021-04-04T09:33:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:33.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: remote image is non-primary 2021-04-04T09:33:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:33:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:33:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:33:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2021-04-04T09:33:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:33:30 2021-04-04T09:33:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+unknown 2021-04-04 09:33:30 2021-04-04T09:33:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:33:30 2021-04-04T09:33:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:33:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:33:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:33:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:33:33.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:33:33.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:33:33.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:33:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:33:30 2021-04-04T09:33:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:33:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+unknown 2021-04-04T09:33:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:33:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+unknown 2021-04-04T09:33:33.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:33:33.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+unknown 2021-04-04T09:33:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster2 mirror test up+unknown 2021-04-04T09:33:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.mirror_status 2021-04-04T09:33:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.mirror_status 2021-04-04T09:33:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror image status mirror/test 2021-04-04T09:33:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:33:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:33:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:33:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:33:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:33:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:32 2021-04-04T09:33:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster1 2021-04-04T09:33:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+unknown 2021-04-04T09:33:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: remote image is non-primary 2021-04-04T09:33:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:30 2021-04-04T09:33:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+unknown /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+unknown 2021-04-04T09:33:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: remote image is non-primary 2021-04-04T09:33:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:33:34.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:33:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:33:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:33:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:34.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:33:32 2021-04-04T09:33:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+unknown 2021-04-04 09:33:32 2021-04-04T09:33:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+unknown 2021-04-04T09:33:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:33:32 2021-04-04T09:33:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror.mirror_status 2021-04-04T09:33:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror.mirror_status 2021-04-04T09:33:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror.mirror_status 2021-04-04T09:33:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2021-04-04T09:33:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:33:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:33:35.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:33:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:33:32 2021-04-04T09:33:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:33:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+unknown 2021-04-04T09:33:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+unknown 2021-04-04T09:33:35.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+unknown 2021-04-04T09:33:35.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+unknown 2021-04-04T09:33:35.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ promote_image cluster2 mirror test 2021-04-04T09:33:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local force= 2021-04-04T09:33:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/test 2021-04-04T09:33:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Image promoted to primary 2021-04-04T09:33:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-04-04T09:33:35.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:35.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-04-04T09:33:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:33:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:33:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:36.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2021-04-04T09:33:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:33:36.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:33:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:33:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:33:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + localgrep cluster_instance=cluster1 2021-04-04T09:33:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: -i state 2021-04-04T09:33:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:33:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:36.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:33:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:33:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:33:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:33:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:36.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:36.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:33:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:33:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:33:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:36.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:36.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:36.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:36.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:33:36.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:33:36.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:33:36.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:33:36.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:33:36.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:33:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:33:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:36.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:36.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:33:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:33:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:33:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:33:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:36.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:36.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:33:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:33:36.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:36.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-04-04T09:33:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:33:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:33:36.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:36.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:33:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:33:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:36.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:33:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:36.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:36.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:36.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:33:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-04-04T09:33:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:33:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:33:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:33:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:33:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:33:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:33:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test 100 2021-04-04T09:33:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:33:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:33:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:33:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:33:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:33:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:33:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:33:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 3125 bytes/sec: 12 MiB/s 2021-04-04T09:33:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2021-04-04T09:33:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:33:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:33:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test 2021-04-04T09:33:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:33:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:33:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:33:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:33:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:33:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test 2021-04-04T09:33:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:33:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:33:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test 2021-04-04T09:33:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2021-04-04T09:33:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:33:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:33:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:33:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2021-04-04T09:33:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:33:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:33:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:33:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:33:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:33:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:33:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:36.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2021-04-04T09:33:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:33:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush mirror/test1 2021-04-04T09:33:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2021-04-04T09:33:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:33:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:33:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:33:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:33:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:33:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:36.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:33:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:36.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2021-04-04T09:33:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:33:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:33:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2021-04-04T09:33:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:33:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:33:37.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:33:37.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:33:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:33:37.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:37.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:33:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:37.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2021-04-04T09:33:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test 2021-04-04T09:33:37.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test 2021-04-04T09:33:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:33:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:33:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:33:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:33:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:33:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:33:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:33:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=24, entry_tid=101], [object_number=0, tag_tid=24, entry_tid=100], [object_number=3, tag_tid=24, entry_tid=99], [object_number=2, tag_tid=24, entry_tid=98]]], state=connected] 2021-04-04T09:33:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=12, entry_tid=101], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:33:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=24, entry_tid=101 2021-04-04T09:33:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test 2021-04-04T09:33:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test ..* 2021-04-04T09:33:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:33:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:33:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:33:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:33:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:33:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:33:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:33:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=24, entry_tid=101], [object_number=0, tag_tid=24, entry_tid=100], [object_number=3, tag_tid=24, entry_tid=99], [object_number=2, tag_tid=24, entry_tid=98]]], state=connected] 2021-04-04T09:33:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=12, entry_tid=101], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:33:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=0, tag_tid=23, entry_tid=0 2021-04-04T09:33:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=24, entry_tid=101 -a object_number=1, tag_tid=24, entry_tid=101 = object_number=0, tag_tid=23, entry_tid=0 2021-04-04T09:33:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=0, tag_tid=23, entry_tid=0 != 2021-04-04T09:33:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:33:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=0, tag_tid=23, entry_tid=0 = 2021-04-04T09:33:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_mirror_pos=object_number=0, tag_tid=23, entry_tid=0 2021-04-04T09:33:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=24, entry_tid=101 2021-04-04T09:33:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo tag_tid=[0-9]* 2021-04-04T09:33:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:33:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_tag=24 2021-04-04T09:33:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=0, tag_tid=23, entry_tid=0 2021-04-04T09:33:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo tag_tid=[0-9]* 2021-04-04T09:33:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:33:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_tag=23 2021-04-04T09:33:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=24, entry_tid=101 2021-04-04T09:33:37.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo entry_tid=[0-9]* 2021-04-04T09:33:37.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:33:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_entry=101 2021-04-04T09:33:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=0, tag_tid=23, entry_tid=0 2021-04-04T09:33:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo entry_tid=[0-9]* 2021-04-04T09:33:37.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:33:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_entry=0 2021-04-04T09:33:37.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 24 = 23 -a 101 -le 0 2021-04-04T09:33:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:33:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:33:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test 2021-04-04T09:33:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:37.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:33:37.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:33:37.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test 2021-04-04T09:33:37.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2021-04-04T09:33:37.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:33:37.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:37.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:33:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:37.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:33:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2021-04-04T09:33:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:33:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:33:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:37.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:37.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:33:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:33:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:33:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:37.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:33:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2021-04-04T09:33:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:33:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush mirror/test1 2021-04-04T09:33:37.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:37.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2021-04-04T09:33:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:33:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:33:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:33:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:33:37.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:33:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:33:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2021-04-04T09:33:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:33:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:33:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2021-04-04T09:33:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:33:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:33:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:33:37.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:33:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:33:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:37.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:37.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:33:37.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2021-04-04T09:33:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test 2021-04-04T09:33:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test 2021-04-04T09:33:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:37.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:33:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:33:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:33:37.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:37.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:33:37.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:33:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:33:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:33:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=24, entry_tid=101], [object_number=0, tag_tid=24, entry_tid=100], [object_number=3, tag_tid=24, entry_tid=99], [object_number=2, tag_tid=24, entry_tid=98]]], state=connected] 2021-04-04T09:33:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=12, entry_tid=101], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:33:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=24, entry_tid=101 2021-04-04T09:33:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test 2021-04-04T09:33:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test ..* 2021-04-04T09:33:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:33:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:33:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:33:37.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:37.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:33:37.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:33:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=24, entry_tid=101], [object_number=0, tag_tid=24, entry_tid=100], [object_number=3, tag_tid=24, entry_tid=99], [object_number=2, tag_tid=24, entry_tid=98]]], state=connected] 2021-04-04T09:33:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=0, tag_tid=23, entry_tid=0], [object_number=1, tag_tid=12, entry_tid=101], [object_number=3, tag_tid=12, entry_tid=99], [object_number=2, tag_tid=12, entry_tid=98]]], state=connected] 2021-04-04T09:33:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:38.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=0, tag_tid=23, entry_tid=0 2021-04-04T09:33:38.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=24, entry_tid=101 -a object_number=1, tag_tid=24, entry_tid=101 = object_number=0, tag_tid=23, entry_tid=0 2021-04-04T09:33:38.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=0, tag_tid=23, entry_tid=0 != object_number=0, tag_tid=23, entry_tid=0 2021-04-04T09:33:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2021-04-04T09:33:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test 2021-04-04T09:33:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:33:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:33:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test 2021-04-04T09:33:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test 2021-04-04T09:33:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:33:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:33:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:33:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test 2021-04-04T09:33:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:33:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:33:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:33:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:33:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:33:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:33:38.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test 2021-04-04T09:33:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 1 closest matches: 2021-04-04T09:33:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush mirror/test1 2021-04-04T09:33:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test 2021-04-04T09:33:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:33:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:33:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:38.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:38.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:33:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:33:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:33:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:38.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:38.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:33:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:38.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test 2021-04-04T09:33:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:33:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:33:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test 2021-04-04T09:33:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:33:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:33:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:33:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:33:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:33:38.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:38.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:38.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:33:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test 2021-04-04T09:33:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test 2021-04-04T09:33:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test 2021-04-04T09:33:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:33:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:33:38.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:33:38.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:33:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:33:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:33:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:33:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=24, entry_tid=101], [object_number=0, tag_tid=24, entry_tid=100], [object_number=3, tag_tid=24, entry_tid=99], [object_number=2, tag_tid=24, entry_tid=98]]], state=connected] 2021-04-04T09:33:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=24, entry_tid=101], [object_number=0, tag_tid=24, entry_tid=100], [object_number=3, tag_tid=24, entry_tid=99], [object_number=2, tag_tid=24, entry_tid=98]]], state=connected] 2021-04-04T09:33:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=24, entry_tid=101 2021-04-04T09:33:38.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test 2021-04-04T09:33:38.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:38.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:38.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test ..* 2021-04-04T09:33:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:33:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.status 2021-04-04T09:33:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.status 2021-04-04T09:33:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test 2021-04-04T09:33:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:33:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:33:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:33:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=24, entry_tid=101], [object_number=0, tag_tid=24, entry_tid=100], [object_number=3, tag_tid=24, entry_tid=99], [object_number=2, tag_tid=24, entry_tid=98]]], state=connected] 2021-04-04T09:33:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=24, entry_tid=101], [object_number=0, tag_tid=24, entry_tid=100], [object_number=3, tag_tid=24, entry_tid=99], [object_number=2, tag_tid=24, entry_tid=98]]], state=connected] 2021-04-04T09:33:38.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.status 2021-04-04T09:33:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=1, tag_tid=24, entry_tid=101 2021-04-04T09:33:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=24, entry_tid=101 -a object_number=1, tag_tid=24, entry_tid=101 = object_number=1, tag_tid=24, entry_tid=101 2021-04-04T09:33:38.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2021-04-04T09:33:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:33:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:33:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying primary_position 2021-04-04T09:33:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:33:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:33:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.mirror_status 2021-04-04T09:33:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.mirror_status 2021-04-04T09:33:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2021-04-04T09:33:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:33:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:33:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:33:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}} 2021-04-04T09:33:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2021-04-04T09:33:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:35 2021-04-04T09:33:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:33:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:33:40.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:37 2021-04-04T09:33:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:33:40.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .*primary_position /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:40.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}} 2021-04-04T09:33:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:33:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:33:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:33:40.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2021-04-04T09:33:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.mirror_status 2021-04-04T09:33:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:33:35 2021-04-04T09:33:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2021-04-04 09:33:35 2021-04-04T09:33:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:33:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:33:35 2021-04-04T09:33:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:33:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:33:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:33:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:33:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:33:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:33:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:33:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:33:35 2021-04-04T09:33:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:33:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:33:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:33:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:33:40.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:33:40.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:40.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test up+stopped 2021-04-04T09:33:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:40.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:40.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:40.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster2 mirror test up+stopped 2021-04-04T09:33:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.mirror_status 2021-04-04T09:33:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.mirror_status 2021-04-04T09:33:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror image status mirror/test 2021-04-04T09:33:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:33:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:33:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:33:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.1 on smithi122 2021-04-04T09:33:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:37 2021-04-04T09:33:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster1 2021-04-04T09:33:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:33:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":1,"entries_per_second":0.0,"non_primary_position":{"entry_tid":0,"object_number":0,"tag_tid":19},"primary_position":{"entry_tid":0,"object_number":0,"tag_tid":23}} 2021-04-04T09:33:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:35 2021-04-04T09:33:41.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+stopped /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:33:41.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: local image is primary 2021-04-04T09:33:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:33:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:33:41.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:33:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:33:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.mirror_status 2021-04-04T09:33:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:33:37 2021-04-04T09:33:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test up+stopped 2021-04-04 09:33:37 2021-04-04T09:33:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:33:37 2021-04-04T09:33:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror.mirror_status 2021-04-04T09:33:41.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror.mirror_status 2021-04-04T09:33:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:41.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror.mirror_status 2021-04-04T09:33:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2021-04-04T09:33:41.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:33:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:33:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:33:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:33:37 2021-04-04T09:33:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:33:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+stopped 2021-04-04T09:33:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:33:41.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+stopped 2021-04-04T09:33:41.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:33:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_images mirror test 2021-04-04T09:33:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:33:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2021-04-04T09:33:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test.export 2021-04-04T09:33:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test.export 2021-04-04T09:33:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export 2021-04-04T09:33:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test.export 2021-04-04T09:33:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test.export 2021-04-04T09:33:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:41.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:33:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:33:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 export mirror/test /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export 2021-04-04T09:33:41.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:33:41.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 export mirror/test /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:33:41.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:33:41.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:33:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test.export 2021-04-04T09:33:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ force_promote_image=test_force_promote 2021-04-04T09:33:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror test_force_promote 2021-04-04T09:33:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2021-04-04T09:33:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test_force_promote 2021-04-04T09:33:41.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test_force_promote 100 2021-04-04T09:33:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:33:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:33:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:33:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:33:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:33:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:33:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:33:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2777.78 bytes/sec: 11 MiB/s 2021-04-04T09:33:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_force_promote 2021-04-04T09:33:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:42.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster2 mirror test_force_promote stopped 2021-04-04T09:33:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=stopped 2021-04-04T09:33:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:33:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:42.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster2 mirror test_force_promote stopped 2021-04-04T09:33:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:33:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:33:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:33:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster2 2021-04-04T09:33:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:33:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster2 2021-04-04T09:33:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:33:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:33:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:43.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:33:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:43.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:33:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:33:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:33:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:33:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:33:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:33:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:33:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:33:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:33:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:43.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:43.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:43.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:33:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:33:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:33:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:33:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:33:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:33:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:33:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:33:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:33:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:33:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:33:43.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:33:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:33:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:33:43.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:43.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:33:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:33:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:33:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:43.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:33:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:43.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:33:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:33:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:33:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:33:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:33:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:33:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:33:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:33:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:33:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:33:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:33:43.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:43.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:33:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:44.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:33:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:33:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:33:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:33:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:33:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:33:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:44.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:33:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:33:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:44.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:44.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:33:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:33:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:33:44.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:33:44.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:33:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:33:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:33:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:44.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:33:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:33:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:33:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:33:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:33:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:33:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:33:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:33:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:33:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:33:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:33:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + echo "state": "Stopped"grep 2021-04-04T09:33:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: -i Replaying 2021-04-04T09:33:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = stopped 2021-04-04T09:33:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test_force_promote 2021-04-04T09:33:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test_force_promote started 2021-04-04T09:33:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:33:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:33:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:45.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test_force_promote started 2021-04-04T09:33:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:33:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:33:45.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:33:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:45.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:33:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + localgrep cluster=cluster1 -i 2021-04-04T09:33:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state 2021-04-04T09:33:45.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:33:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:33:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:33:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:33:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:33:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:33:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:33:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:45.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:33:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:46.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:46.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:46.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:46.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:33:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:33:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:33:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:33:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:33:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:46.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:33:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:46.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:33:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:46.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:33:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:46.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:46.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:33:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:33:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:46.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:33:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:46.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:33:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:33:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:33:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:33:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:33:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:33:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:33:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:33:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:33:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:33:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:46.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:33:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:33:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:46.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:33:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:33:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:33:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:33:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:33:46.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:33:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:33:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:33:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:33:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:33:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_force_promote 2021-04-04T09:33:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:33:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:33:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test_force_promote 2021-04-04T09:33:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:33:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:33:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:33:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:33:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:33:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test_force_promote 2021-04-04T09:33:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:33:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_force_promote ] 2021-04-04T09:33:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test_force_promote 2021-04-04T09:33:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_force_promote 2021-04-04T09:33:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:33:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:46.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:33:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:33:46.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_force_promote 2021-04-04T09:33:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:33:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:33:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:46.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:33:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:33:46.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:33:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:46.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:33:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_force_promote 2021-04-04T09:33:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:33:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_force_promote 2021-04-04T09:33:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:33:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:33:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:33:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:33:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:33:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:33:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_force_promote 2021-04-04T09:33:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:33:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test_force_promote 2021-04-04T09:33:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:33:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:33:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:33:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:33:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:33:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:33:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test_force_promote 2021-04-04T09:33:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:33:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test_force_promote 2021-04-04T09:33:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:33:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:33:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:33:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:33:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:33:46.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:46.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:33:46.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:33:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:33:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:33:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:33:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:33:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:33:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:33:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:33:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:33:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test_force_promote 2021-04-04T09:33:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test_force_promote 2021-04-04T09:33:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test_force_promote 2021-04-04T09:33:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:33:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_force_promote.status 2021-04-04T09:33:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_force_promote.status 2021-04-04T09:33:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.status 2021-04-04T09:33:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test_force_promote 2021-04-04T09:33:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.status 2021-04-04T09:33:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:33:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:33:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:33:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:33:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:33:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.status 2021-04-04T09:33:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=3, tag_tid=2, entry_tid=3 2021-04-04T09:33:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test_force_promote 2021-04-04T09:33:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:47.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test_force_promote ..* 2021-04-04T09:33:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:47.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:33:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_force_promote.status 2021-04-04T09:33:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_force_promote.status 2021-04-04T09:33:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.status 2021-04-04T09:33:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + rbd --cluster cluster2 journal status --imagetee mirror/test_force_promote /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.status 2021-04-04T09:33:47.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2021-04-04T09:33:47.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:33:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:33:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:33:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:33:47.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:33:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.status 2021-04-04T09:33:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=3, tag_tid=2, entry_tid=3 2021-04-04T09:33:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=3, tag_tid=2, entry_tid=3 -a object_number=3, tag_tid=2, entry_tid=3 = object_number=3, tag_tid=2, entry_tid=3 2021-04-04T09:33:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_force_promote up+replaying primary_position 2021-04-04T09:33:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:33:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:33:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:48.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test_force_promote up+replaying primary_position 2021-04-04T09:33:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:48.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:33:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:33:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test_force_promote.mirror_status 2021-04-04T09:33:48.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test_force_promote.mirror_status 2021-04-04T09:33:48.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_force_promote.mirror_status 2021-04-04T09:33:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test_force_promote 2021-04-04T09:33:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_force_promote.mirror_status 2021-04-04T09:33:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test_force_promote: 2021-04-04T09:33:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T09:33:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:33:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:33:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.3 on smithi122 2021-04-04T09:33:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:46 2021-04-04T09:33:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:33:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:33:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:47 2021-04-04T09:33:48.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_force_promote.mirror_status 2021-04-04T09:33:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:33:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .*primary_position /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_force_promote.mirror_status 2021-04-04T09:33:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:33:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:33:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:33:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:33:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_force_promote.mirror_status 2021-04-04T09:33:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2021-04-04T09:33:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_force_promote.mirror_status 2021-04-04T09:33:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:33:46 2021-04-04T09:33:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_force_promote up+replaying 2021-04-04 09:33:46 2021-04-04T09:33:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:33:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:33:46 2021-04-04T09:33:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:33:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:33:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:33:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:33:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:33:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:33:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_force_promote']/last_update 2021-04-04T09:33:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:33:46 2021-04-04T09:33:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_force_promote']/state 2021-04-04T09:33:48.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:33:48.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:33:48.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:33:48.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:33:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_force_promote up+stopped 2021-04-04T09:33:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster2 mirror test_force_promote up+stopped 2021-04-04T09:33:49.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_force_promote.mirror_status 2021-04-04T09:33:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_force_promote.mirror_status 2021-04-04T09:33:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:49.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.mirror_status 2021-04-04T09:33:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror image status mirror/test_force_promote 2021-04-04T09:33:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.mirror_status 2021-04-04T09:33:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test_force_promote: 2021-04-04T09:33:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T09:33:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:33:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2021-04-04T09:33:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:47 2021-04-04T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster1 2021-04-04T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:33:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:46 2021-04-04T09:33:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+stopped /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.mirror_status 2021-04-04T09:33:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:33:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.mirror_status 2021-04-04T09:33:49.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: local image is primary 2021-04-04T09:33:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + grep ^up+ 2021-04-04T09:33:49.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:echo up+stopped 2021-04-04T09:33:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:33:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.mirror_status 2021-04-04T09:33:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2021-04-04T09:33:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.mirror_status 2021-04-04T09:33:49.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:33:47 2021-04-04T09:33:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_force_promote up+stopped 2021-04-04 09:33:47 2021-04-04T09:33:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:49.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:33:47 2021-04-04T09:33:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror.mirror_status 2021-04-04T09:33:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror.mirror_status 2021-04-04T09:33:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror.mirror_status 2021-04-04T09:33:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2021-04-04T09:33:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:33:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:33:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_force_promote']/last_update 2021-04-04T09:33:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:33:47 2021-04-04T09:33:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_force_promote']/state 2021-04-04T09:33:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+stopped 2021-04-04T09:33:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:33:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+stopped 2021-04-04T09:33:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:33:49.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ promote_image cluster1 mirror test_force_promote --force 2021-04-04T09:33:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local force=--force 2021-04-04T09:33:49.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 mirror image promote mirror/test_force_promote --force 2021-04-04T09:33:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Image promoted to primary 2021-04-04T09:33:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_stopped cluster1 mirror test_force_promote 2021-04-04T09:33:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test_force_promote stopped 2021-04-04T09:33:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=stopped 2021-04-04T09:33:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:33:49.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test_force_promote stopped 2021-04-04T09:33:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:33:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:33:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:33:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:33:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1+ 2021-04-04T09:33:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:33:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i+ statesleep 2021-04-04T09:33:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 0 2021-04-04T09:33:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:33:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:33:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:33:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:50.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:50.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:50.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:50.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:50.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:33:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:33:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:33:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:50.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:33:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:33:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:33:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:50.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:33:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:33:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:33:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:50.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:33:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:33:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:33:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:33:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:33:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:33:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:33:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:51.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:33:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:51.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:33:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:33:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:51.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:33:51.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:51.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:51.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:33:51.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:51.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:33:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:33:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:51.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:33:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:33:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:51.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:51.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:51.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:33:51.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:51.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:33:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:33:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:33:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:33:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:33:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:51.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:33:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Stopped" 2021-04-04T09:33:51.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:33:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = stopped 2021-04-04T09:33:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_stopped cluster2 mirror test_force_promote 2021-04-04T09:33:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster2 mirror test_force_promote stopped 2021-04-04T09:33:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=stopped 2021-04-04T09:33:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:33:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster2 mirror test_force_promote stopped 2021-04-04T09:33:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=stopped 2021-04-04T09:33:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:33:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:33:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster2 2021-04-04T09:33:52.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep+ -ishift state 2021-04-04T09:33:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2021-04-04T09:33:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster2 2021-04-04T09:33:52.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:33:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:33:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:33:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:33:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:33:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:33:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:33:52.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:52.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:33:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:52.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:33:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:52.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:33:52.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:52.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:33:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:33:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:33:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:33:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:33:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:33:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:33:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:33:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:52.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:33:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_force_promote'] 2021-04-04T09:33:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status mirror/test_force_promote 2021-04-04T09:33:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:33:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:33:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:33:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:33:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:33:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:33:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:33:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:33:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:33:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:33:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:33:52.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:52.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:33:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:33:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:33:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:33:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status mirror/test_force_promote 2021-04-04T09:33:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Stopped" 2021-04-04T09:33:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Stopped" 2021-04-04T09:33:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:33:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test stopped = stopped 2021-04-04T09:33:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_force_promote up+stopped 2021-04-04T09:33:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:52.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test_force_promote up+stopped 2021-04-04T09:33:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test_force_promote.mirror_status 2021-04-04T09:33:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test_force_promote.mirror_status 2021-04-04T09:33:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:53.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_force_promote.mirror_status 2021-04-04T09:33:53.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test_force_promote 2021-04-04T09:33:53.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_force_promote.mirror_status 2021-04-04T09:33:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test_force_promote: 2021-04-04T09:33:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T09:33:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:33:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.3 on smithi122 2021-04-04T09:33:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:50 2021-04-04T09:33:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:53.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:33:53.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:53.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:33:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:52 2021-04-04T09:33:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+stopped /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_force_promote.mirror_status 2021-04-04T09:33:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:33:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_force_promote.mirror_status 2021-04-04T09:33:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: local image is primary 2021-04-04T09:33:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:33:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:33:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:33:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_force_promote.mirror_status 2021-04-04T09:33:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2021-04-04T09:33:53.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_force_promote.mirror_status 2021-04-04T09:33:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:33:50 2021-04-04T09:33:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_force_promote up+stopped 2021-04-04 09:33:50 2021-04-04T09:33:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:33:50 2021-04-04T09:33:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:33:53.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:33:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:33:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:33:53.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:33:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:33:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_force_promote']/last_update 2021-04-04T09:33:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:33:50 2021-04-04T09:33:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_force_promote']/state 2021-04-04T09:33:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+stopped 2021-04-04T09:33:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:33:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+stopped 2021-04-04T09:33:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:33:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster2 mirror test_force_promote up+stopped 2021-04-04T09:33:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster2 mirror test_force_promote up+stopped 2021-04-04T09:33:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern= 2021-04-04T09:33:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:33:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_force_promote.mirror_status 2021-04-04T09:33:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_force_promote.mirror_status 2021-04-04T09:33:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.mirror_status 2021-04-04T09:33:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.mirror_status 2021-04-04T09:33:54.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:CEPH_ARGS= rbd --cluster cluster2 mirror image status mirror/test_force_promote 2021-04-04T09:33:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test_force_promote: 2021-04-04T09:33:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T09:33:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:33:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.2 on smithi122 2021-04-04T09:33:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:52 2021-04-04T09:33:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:33:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster1 2021-04-04T09:33:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:33:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:33:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:33:50 2021-04-04T09:33:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+stopped /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.mirror_status 2021-04-04T09:33:54.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:33:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .* /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.mirror_status 2021-04-04T09:33:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: local image is primary 2021-04-04T09:33:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+stopped 2021-04-04T09:33:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:33:54.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:33:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.mirror_status 2021-04-04T09:33:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2021-04-04T09:33:54.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_force_promote.mirror_status 2021-04-04T09:33:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:33:52 2021-04-04T09:33:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster2 mirror test_force_promote up+stopped 2021-04-04 09:33:52 2021-04-04T09:33:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+stopped 2021-04-04T09:33:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:33:52 2021-04-04T09:33:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror.mirror_status 2021-04-04T09:33:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror.mirror_status 2021-04-04T09:33:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:33:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror.mirror_status 2021-04-04T09:33:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2021-04-04T09:33:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:33:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:33:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_force_promote']/last_update 2021-04-04T09:33:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:33:52 2021-04-04T09:33:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_force_promote']/state 2021-04-04T09:33:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+stopped 2021-04-04T09:33:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + grep up+stopped 2021-04-04T09:33:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:echo up+stopped 2021-04-04T09:33:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+stopped 2021-04-04T09:33:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:33:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster1 mirror test_force_promote 100 2021-04-04T09:33:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:33:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:33:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:33:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:33:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:33:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:33:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:33:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 3125 bytes/sec: 12 MiB/s 2021-04-04T09:33:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test_force_promote 100 2021-04-04T09:33:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:55.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_force_promote 2021-04-04T09:33:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:33:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:33:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:33:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:33:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test_force_promote --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:33:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:33:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:33:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2500 bytes/sec: 9.8 MiB/s 2021-04-04T09:33:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog TEST: cloned images 2021-04-04T09:33:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:33:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:33:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:33:55 TEST: cloned images 2021-04-04T09:33:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:33:55 TEST: cloned images 2021-04-04T09:33:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog - default 2021-04-04T09:33:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:33:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:33:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:33:55 - default 2021-04-04T09:33:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:33:55 - default 2021-04-04T09:33:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ parent_image=test_parent 2021-04-04T09:33:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ parent_snap=snap 2021-04-04T09:33:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror_parent test_parent 2021-04-04T09:33:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:33:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:33:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2021-04-04T09:33:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:33:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror_parent/test_parent 2021-04-04T09:33:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror_parent test_parent 100 2021-04-04T09:33:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:33:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:33:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:33:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:33:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:33:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:33:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror_parent/test_parent --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:33:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:33:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:33:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2500 bytes/sec: 9.8 MiB/s 2021-04-04T09:33:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap 2021-04-04T09:33:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:33:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:33:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap 2021-04-04T09:33:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap 2021-04-04T09:33:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-04-04T09:33:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ protect_snapshot cluster2 mirror_parent test_parent snap 2021-04-04T09:33:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:33:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:33:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap 2021-04-04T09:33:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap protect mirror_parent/test_parent@snap 2021-04-04T09:33:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clone_image=test_clone 2021-04-04T09:33:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clone_image cluster2 mirror_parent test_parent snap mirror test_clone 2021-04-04T09:33:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local parent_pool=mirror_parent 2021-04-04T09:33:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local parent_image=test_parent 2021-04-04T09:33:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local parent_snap=snap 2021-04-04T09:33:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local clone_pool=mirror 2021-04-04T09:33:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local clone_image=test_clone 2021-04-04T09:33:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 6 2021-04-04T09:33:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap mirror/test_clone --image-feature layering,exclusive-lock,journaling 2021-04-04T09:33:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test_clone 100 2021-04-04T09:33:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:33:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone 2021-04-04T09:33:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:33:56.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:33:56.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:33:56.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:33:56.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test_clone --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:33:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:33:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:33:56.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 294.118 bytes/sec: 1.1 MiB/s 2021-04-04T09:33:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror_parent test_parent journal 2021-04-04T09:33:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:33:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:33:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:33:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=journal 2021-04-04T09:33:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror_parent/test_parent journal 2021-04-04T09:33:56.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2021-04-04T09:33:56.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror_parent test_parent 2021-04-04T09:33:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:33:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:33:56.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror_parent test_parent started 2021-04-04T09:33:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:33:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:33:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:33:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:33:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:57.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror_parent test_parent started 2021-04-04T09:33:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:33:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:33:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:33:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:33:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:33:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:33:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:33:57.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:33:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:33:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:33:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:33:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:33:57.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:33:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:57.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:33:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:33:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:33:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:33:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:57.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2021-04-04T09:33:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:33:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:33:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:33:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:33:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:33:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:33:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2021-04-04T09:33:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:33:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:33:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:33:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:33:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:33:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:57.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:57.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:33:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2021-04-04T09:33:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:33:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:33:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:33:58.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:33:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:33:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:33:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:33:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:33:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:33:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:33:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:33:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:58.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:33:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:33:58.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:33:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:33:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2021-04-04T09:33:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:33:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:33:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:33:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:33:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:33:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:59.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:59.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:59.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:59.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:59.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:33:59.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:33:59.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:59.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:33:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:33:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:33:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:33:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:33:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:33:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:33:59.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:59.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2021-04-04T09:33:59.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:59.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:33:59.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:59.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:59.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:33:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:59.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:59.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:59.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:59.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:33:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:33:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:33:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:33:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:33:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:33:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:33:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:33:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:33:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2021-04-04T09:33:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:33:59.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:33:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:33:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:33:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:33:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:33:59.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:33:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:33:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:33:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:33:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:33:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2021-04-04T09:33:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:59.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:33:59.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:33:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:33:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:33:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:33:59.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:33:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:33:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:33:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:33:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:33:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:33:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:33:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:33:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:33:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:33:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:33:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:33:59.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:33:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:33:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:33:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:33:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:33:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:33:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:33:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:33:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:33:59.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror_parent/test_parent 2021-04-04T09:33:59.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror_parent/test_parent'] 2021-04-04T09:33:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:33:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:33:59.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:34:01.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:01.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror_parent/test_parent 2021-04-04T09:34:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror_parent/test_parent 2021-04-04T09:34:01.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:01.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:34:01.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:34:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:34:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:34:01.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:34:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:34:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror_parent test_parent 2021-04-04T09:34:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:34:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:01.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:01.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:01.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:34:01.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror_parent test_parent 2021-04-04T09:34:01.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:34:01.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:01.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:01.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:34:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:34:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:34:01.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:34:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror_parent test_parent 2021-04-04T09:34:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:34:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_parent ] 2021-04-04T09:34:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror_parent/test_parent 2021-04-04T09:34:01.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror_parent/test_parent 2021-04-04T09:34:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:34:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:34:01.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:34:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:34:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror_parent/test_parent 2021-04-04T09:34:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:01.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:01.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:01.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:01.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:02.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:02.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:02.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:02.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror_parent/test_parent 2021-04-04T09:34:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror_parent test_parent 2021-04-04T09:34:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror_parent test_parent 2021-04-04T09:34:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:02.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:34:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror_parent/test_parent 2021-04-04T09:34:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:34:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:34:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:34:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2021-04-04T09:34:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=1cd21fe3-be3b-4a58-ac2d-a25de318debc, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected] 2021-04-04T09:34:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=3, tag_tid=4, entry_tid=3 2021-04-04T09:34:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror_parent test_parent 2021-04-04T09:34:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror_parent test_parent ..* 2021-04-04T09:34:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:02.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:34:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror_parent/test_parent 2021-04-04T09:34:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:34:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:34:02.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:34:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2021-04-04T09:34:02.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=1cd21fe3-be3b-4a58-ac2d-a25de318debc, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected] 2021-04-04T09:34:02.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=1, tag_tid=4, entry_tid=1 2021-04-04T09:34:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=3, tag_tid=4, entry_tid=3 -a object_number=3, tag_tid=4, entry_tid=3 = object_number=1, tag_tid=4, entry_tid=1 2021-04-04T09:34:02.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=1, tag_tid=4, entry_tid=1 != 2021-04-04T09:34:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:34:02.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=1, tag_tid=4, entry_tid=1 = 2021-04-04T09:34:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_mirror_pos=object_number=1, tag_tid=4, entry_tid=1 2021-04-04T09:34:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=3, tag_tid=4, entry_tid=3 2021-04-04T09:34:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo tag_tid=[0-9]* 2021-04-04T09:34:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:34:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_tag=4 2021-04-04T09:34:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=4, entry_tid=1 2021-04-04T09:34:02.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo tag_tid=[0-9]* 2021-04-04T09:34:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:34:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_tag=4 2021-04-04T09:34:02.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=3, tag_tid=4, entry_tid=3 2021-04-04T09:34:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo entry_tid=[0-9]* 2021-04-04T09:34:02.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:34:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_entry=3 2021-04-04T09:34:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=4, entry_tid=1 2021-04-04T09:34:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo entry_tid=[0-9]* 2021-04-04T09:34:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:34:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_entry=1 2021-04-04T09:34:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 4 = 4 -a 3 -le 1 2021-04-04T09:34:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:34:02.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:34:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror_parent test_parent 2021-04-04T09:34:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:02.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:34:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_parent ] 2021-04-04T09:34:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror_parent/test_parent 2021-04-04T09:34:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror_parent/test_parent 2021-04-04T09:34:02.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:34:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:02.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:34:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:34:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:34:02.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror_parent/test_parent 2021-04-04T09:34:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:02.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:02.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:02.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror_parent/test_parent 2021-04-04T09:34:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror_parent test_parent 2021-04-04T09:34:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror_parent test_parent 2021-04-04T09:34:02.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:34:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror_parent/test_parent 2021-04-04T09:34:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:34:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:34:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:34:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2021-04-04T09:34:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=1cd21fe3-be3b-4a58-ac2d-a25de318debc, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected] 2021-04-04T09:34:02.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=3, tag_tid=4, entry_tid=3 2021-04-04T09:34:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror_parent test_parent 2021-04-04T09:34:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror_parent test_parent ..* 2021-04-04T09:34:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:34:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status+ --image mirror_parent/test_parent 2021-04-04T09:34:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:34:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:34:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:34:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2021-04-04T09:34:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=1cd21fe3-be3b-4a58-ac2d-a25de318debc, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=99], [object_number=2, tag_tid=1, entry_tid=98]]], state=connected] 2021-04-04T09:34:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=1, tag_tid=4, entry_tid=1 2021-04-04T09:34:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=3, tag_tid=4, entry_tid=3 -a object_number=3, tag_tid=4, entry_tid=3 = object_number=1, tag_tid=4, entry_tid=1 2021-04-04T09:34:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=1, tag_tid=4, entry_tid=1 != object_number=1, tag_tid=4, entry_tid=1 2021-04-04T09:34:02.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2021-04-04T09:34:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror_parent test_parent 2021-04-04T09:34:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_parent ] 2021-04-04T09:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror_parent/test_parent 2021-04-04T09:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror_parent/test_parent 2021-04-04T09:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:34:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:34:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:34:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:34:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror_parent/test_parent 2021-04-04T09:34:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:03.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:03.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:03.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:03.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:03.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:03.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:03.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:03.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:03.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:03.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:03.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:03.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:03.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:03.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:03.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror_parent/test_parent 2021-04-04T09:34:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror_parent test_parent 2021-04-04T09:34:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:03.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror_parent test_parent 2021-04-04T09:34:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:34:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + sed -e s|[/ ]|_|g 2021-04-04T09:34:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:echo cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:03.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee+ /tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd --cluster cluster2 journal status --image mirror_parent/test_parent 2021-04-04T09:34:03.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:34:03.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:34:03.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:34:03.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2021-04-04T09:34:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=1cd21fe3-be3b-4a58-ac2d-a25de318debc, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2021-04-04T09:34:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=3, tag_tid=4, entry_tid=3 2021-04-04T09:34:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror_parent test_parent 2021-04-04T09:34:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:03.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:03.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror_parent test_parent ..* 2021-04-04T09:34:03.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:03.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:03.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:03.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:34:03.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:03.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:03.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:03.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror_parent/test_parent 2021-04-04T09:34:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:34:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:34:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:34:03.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2021-04-04T09:34:03.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=1cd21fe3-be3b-4a58-ac2d-a25de318debc, commit_position=[positions=[[object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2], [object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0]]], state=connected] 2021-04-04T09:34:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.status 2021-04-04T09:34:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=3, tag_tid=4, entry_tid=3 2021-04-04T09:34:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=3, tag_tid=4, entry_tid=3 -a object_number=3, tag_tid=4, entry_tid=3 = object_number=3, tag_tid=4, entry_tid=3 2021-04-04T09:34:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:03.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror_parent test_parent up+replaying primary_position 2021-04-04T09:34:03.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:03.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:03.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:03.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:34:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:34:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:34:03.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror_parent test_parent up+replaying primary_position 2021-04-04T09:34:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:34:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:34:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:34:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror_parent-test_parent.mirror_status 2021-04-04T09:34:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror_parent-test_parent.mirror_status 2021-04-04T09:34:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:04.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror_parent-test_parent.mirror_status 2021-04-04T09:34:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror_parent-test_parent.mirror_status 2021-04-04T09:34:04.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror_parent/test_parent 2021-04-04T09:34:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test_parent: 2021-04-04T09:34:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: ed3379f2-3a26-4e23-8dd8-e2567a8361ce 2021-04-04T09:34:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:34:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}} 2021-04-04T09:34:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2021-04-04T09:34:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:34:01 2021-04-04T09:34:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:34:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:34:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:34:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:34:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:34:02 2021-04-04T09:34:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror_parent-test_parent.mirror_status 2021-04-04T09:34:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:34:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .*primary_position /tmp/tmp.zj6T3cKcnO/cluster1-mirror_parent-test_parent.mirror_status 2021-04-04T09:34:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":3},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":4}} 2021-04-04T09:34:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:34:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:34:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:34:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:34:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror_parent-test_parent.mirror_status 2021-04-04T09:34:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:34:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror_parent-test_parent.mirror_status 2021-04-04T09:34:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:34:01 2021-04-04T09:34:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror_parent test_parent up+replaying 2021-04-04 09:34:01 2021-04-04T09:34:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:34:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:34:01 2021-04-04T09:34:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror_parent.mirror_status 2021-04-04T09:34:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror_parent.mirror_status 2021-04-04T09:34:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror_parent.mirror_status 2021-04-04T09:34:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror_parent --verbose --format xml 2021-04-04T09:34:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:34:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:34:04.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_parent']/last_update 2021-04-04T09:34:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:34:01 2021-04-04T09:34:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_parent']/state 2021-04-04T09:34:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:34:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:34:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:34:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:34:04.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_images mirror_parent test_parent 2021-04-04T09:34:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:04.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2021-04-04T09:34:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror_parent-test_parent.export 2021-04-04T09:34:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror_parent-test_parent.export 2021-04-04T09:34:04.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.export 2021-04-04T09:34:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror_parent-test_parent.export 2021-04-04T09:34:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror_parent-test_parent.export 2021-04-04T09:34:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:04.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.zj6T3cKcnO/cluster1-mirror_parent-test_parent.export 2021-04-04T09:34:04.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror_parent-test_parent.export 2021-04-04T09:34:04.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 export mirror_parent/test_parent /tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.export 2021-04-04T09:34:04.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:34:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 export mirror_parent/test_parent /tmp/tmp.zj6T3cKcnO/cluster1-mirror_parent-test_parent.export 2021-04-04T09:34:04.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:34:04.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror_parent-test_parent.export 2021-04-04T09:34:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror_parent-test_parent.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror_parent-test_parent.export 2021-04-04T09:34:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone 2021-04-04T09:34:05.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone 2021-04-04T09:34:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone started 2021-04-04T09:34:05.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone 2021-04-04T09:34:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:34:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:34:05.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test_clone started 2021-04-04T09:34:06.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone 2021-04-04T09:34:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:34:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:34:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:34:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test_clone 2021-04-04T09:34:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:34:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local instance=cluster1 2021-04-04T09:34:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:34:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:34:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:34:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_clone 2021-04-04T09:34:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:06.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone 2021-04-04T09:34:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone'] 2021-04-04T09:34:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_clone 2021-04-04T09:34:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:34:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:34:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:06.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:34:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:34:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:34:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:34:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone 2021-04-04T09:34:06.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone'] 2021-04-04T09:34:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test_clone 2021-04-04T09:34:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:34:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:34:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:06.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:06.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:34:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:34:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:34:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:06.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:34:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:06.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone 2021-04-04T09:34:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone'] 2021-04-04T09:34:06.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test_clone 2021-04-04T09:34:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:34:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:34:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:34:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:34:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:34:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:34:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:06.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:06.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone 2021-04-04T09:34:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone'] 2021-04-04T09:34:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:07.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_clone 2021-04-04T09:34:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:07.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:07.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone 2021-04-04T09:34:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone'] 2021-04-04T09:34:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_clone 2021-04-04T09:34:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:34:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:34:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:34:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:34:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:34:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:34:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone 2021-04-04T09:34:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone'] 2021-04-04T09:34:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:07.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test_clone 2021-04-04T09:34:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:34:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:07.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:34:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:34:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:34:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:34:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:07.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:07.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:34:07.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone 2021-04-04T09:34:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone'] 2021-04-04T09:34:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test_clone 2021-04-04T09:34:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:34:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:34:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:07.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:34:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:34:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:34:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:34:07.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone 2021-04-04T09:34:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone'] 2021-04-04T09:34:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:08.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:34:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:10.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_clone 2021-04-04T09:34:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:10.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:10.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:10.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:10.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:10.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone 2021-04-04T09:34:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:34:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:34:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:34:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:34:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:34:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:34:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_clone 2021-04-04T09:34:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:34:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone 2021-04-04T09:34:10.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:34:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test_clone 2021-04-04T09:34:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:34:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone 2021-04-04T09:34:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:34:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:34:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:34:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:34:10.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test_clone 2021-04-04T09:34:10.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:10.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone 2021-04-04T09:34:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:34:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_clone ] 2021-04-04T09:34:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test_clone 2021-04-04T09:34:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_clone 2021-04-04T09:34:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:34:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:34:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:34:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:34:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_clone 2021-04-04T09:34:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:10.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:10.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:10.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_clone 2021-04-04T09:34:10.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:10.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test_clone 2021-04-04T09:34:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone 2021-04-04T09:34:10.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test_clone 2021-04-04T09:34:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone 2021-04-04T09:34:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:34:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_clone.status 2021-04-04T09:34:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_clone.status 2021-04-04T09:34:10.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_clone.status 2021-04-04T09:34:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_clone.status+ 2021-04-04T09:34:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd --cluster cluster2 journal status --image mirror/test_clone 2021-04-04T09:34:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:34:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:34:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:34:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:34:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:34:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_clone.status 2021-04-04T09:34:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=3, tag_tid=2, entry_tid=3 2021-04-04T09:34:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test_clone 2021-04-04T09:34:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone 2021-04-04T09:34:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test_clone ..* 2021-04-04T09:34:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone 2021-04-04T09:34:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:34:10.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_clone.status 2021-04-04T09:34:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_clone.status 2021-04-04T09:34:10.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_clone.status 2021-04-04T09:34:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test_clone 2021-04-04T09:34:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_clone.status 2021-04-04T09:34:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:34:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:34:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:34:10.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:34:10.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:34:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_clone.status 2021-04-04T09:34:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=3, tag_tid=2, entry_tid=3 2021-04-04T09:34:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=3, tag_tid=2, entry_tid=3 -a object_number=3, tag_tid=2, entry_tid=3 = object_number=3, tag_tid=2, entry_tid=3 2021-04-04T09:34:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_clone up+replaying primary_position 2021-04-04T09:34:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:10.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone 2021-04-04T09:34:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:34:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:34:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:34:10.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test_clone up+replaying primary_position 2021-04-04T09:34:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone 2021-04-04T09:34:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:34:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:34:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:34:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test_clone.mirror_status 2021-04-04T09:34:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test_clone.mirror_status 2021-04-04T09:34:11.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_clone.mirror_status 2021-04-04T09:34:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test_clone 2021-04-04T09:34:11.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_clone.mirror_status 2021-04-04T09:34:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test_clone: 2021-04-04T09:34:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: a9fc372c-84b2-41cd-a172-f5e0c63e6d90 2021-04-04T09:34:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:34:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:34:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2021-04-04T09:34:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:34:08 2021-04-04T09:34:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:34:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:34:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:34:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:34:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:34:07 2021-04-04T09:34:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_clone.mirror_status 2021-04-04T09:34:11.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:34:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .*primary_position /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_clone.mirror_status 2021-04-04T09:34:11.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:34:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:34:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:34:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:34:11.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:34:11.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_clone.mirror_status 2021-04-04T09:34:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:34:11.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_clone.mirror_status 2021-04-04T09:34:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:34:08 2021-04-04T09:34:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_clone up+replaying 2021-04-04 09:34:08 2021-04-04T09:34:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone 2021-04-04T09:34:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:34:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:34:08 2021-04-04T09:34:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:34:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:34:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:34:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:34:11.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:34:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:34:11.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_clone']/last_update 2021-04-04T09:34:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:34:08 2021-04-04T09:34:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_clone']/state 2021-04-04T09:34:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:34:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:34:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:34:11.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:34:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:11.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_images mirror test_clone 2021-04-04T09:34:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone 2021-04-04T09:34:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2021-04-04T09:34:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_clone.export 2021-04-04T09:34:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_clone.export 2021-04-04T09:34:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_clone.export 2021-04-04T09:34:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test_clone.export 2021-04-04T09:34:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test_clone.export 2021-04-04T09:34:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_clone.export 2021-04-04T09:34:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_clone.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_clone.export 2021-04-04T09:34:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 export mirror/test_clone /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_clone.export 2021-04-04T09:34:12.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:34:12.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 export mirror/test_clone /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_clone.export 2021-04-04T09:34:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:34:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_clone.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_clone.export 2021-04-04T09:34:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_clone.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_clone.export 2021-04-04T09:34:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog - clone v1 2021-04-04T09:34:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:34:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:34:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:34:12 - clone v1 2021-04-04T09:34:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:34:12 - clone v1 2021-04-04T09:34:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clone_image cluster1 mirror_parent test_parent snap mirror test_clone1 2021-04-04T09:34:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local parent_pool=mirror_parent 2021-04-04T09:34:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local parent_image=test_parent 2021-04-04T09:34:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local parent_snap=snap 2021-04-04T09:34:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local clone_pool=mirror 2021-04-04T09:34:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local clone_image=test_clone1 2021-04-04T09:34:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 6 2021-04-04T09:34:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 clone mirror_parent/test_parent@snap mirror/test_clone1 --image-feature layering,exclusive-lock,journaling 2021-04-04T09:34:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clone_image cluster2 mirror_parent test_parent snap mirror test_clone_v1 --rbd-default-clone-format 1 2021-04-04T09:34:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:12.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local parent_pool=mirror_parent 2021-04-04T09:34:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local parent_image=test_parent 2021-04-04T09:34:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local parent_snap=snap 2021-04-04T09:34:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local clone_pool=mirror 2021-04-04T09:34:12.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local clone_image=test_clone_v1 2021-04-04T09:34:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 6 2021-04-04T09:34:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap mirror/test_clone_v1 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 1 2021-04-04T09:34:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_clone_format cluster2 mirror test_clone_v1 2021-04-04T09:34:12.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v1 2021-04-04T09:34:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 info mirror/test_clone_v1 2021-04-04T09:34:12.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk BEGIN { 2021-04-04T09:34:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: format = 1 2021-04-04T09:34:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: } 2021-04-04T09:34:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: $1 == "parent:" { 2021-04-04T09:34:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: parent = $2 2021-04-04T09:34:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: } 2021-04-04T09:34:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: /op_features: .*clone-child/ { 2021-04-04T09:34:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: format = 2 2021-04-04T09:34:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: } 2021-04-04T09:34:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: END { 2021-04-04T09:34:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: if (!parent) exit 1 2021-04-04T09:34:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: print format 2021-04-04T09:34:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: } 2021-04-04T09:34:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 1 = 1 2021-04-04T09:34:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone_v1 2021-04-04T09:34:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v1 2021-04-04T09:34:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone_v1 started 2021-04-04T09:34:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v1 2021-04-04T09:34:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:34:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:34:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test_clone_v1 started 2021-04-04T09:34:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v1 2021-04-04T09:34:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:34:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:34:13.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:34:13.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:13.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:34:13.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + shift 2021-04-04T09:34:13.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:34:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:34:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:34:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:34:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:13.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:13.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:13.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:13.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2021-04-04T09:34:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:34:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:34:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:34:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:34:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:34:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:34:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:13.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2021-04-04T09:34:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:34:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:34:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:14.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:14.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:14.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:34:14.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:34:14.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:34:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:14.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:14.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:34:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:14.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2021-04-04T09:34:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:14.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:14.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:34:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:34:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:34:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:34:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:34:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:34:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2021-04-04T09:34:14.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:15.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:15.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:15.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:15.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:15.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:15.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:15.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:15.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2021-04-04T09:34:15.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:15.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:34:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:15.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:34:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:34:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:34:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:34:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:34:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2021-04-04T09:34:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:34:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:34:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:34:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:34:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:34:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:34:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:15.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:15.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2021-04-04T09:34:15.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:15.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:34:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:34:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:15.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:15.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:15.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:15.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:34:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:34:15.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:15.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:34:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:15.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:15.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:15.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:34:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:15.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:15.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2021-04-04T09:34:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:34:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:17.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:17.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:17.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:17.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:17.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:17.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:17.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:17.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2021-04-04T09:34:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:17.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:17.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:34:17.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:17.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:34:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:17.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:34:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:34:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:34:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:17.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:17.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:17.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:34:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v1'] 2021-04-04T09:34:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:34:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:34:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:34:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:34:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:34:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:18.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:18.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:34:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:18.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v1 2021-04-04T09:34:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:34:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:34:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:34:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:34:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:34:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:34:18.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_clone_format cluster1 mirror test_clone_v1 2021-04-04T09:34:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v1 2021-04-04T09:34:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test_clone_v1 2021-04-04T09:34:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk BEGIN { 2021-04-04T09:34:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: format = 1 2021-04-04T09:34:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: } 2021-04-04T09:34:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: $1 == "parent:" { 2021-04-04T09:34:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: parent = $2 2021-04-04T09:34:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: } 2021-04-04T09:34:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: /op_features: .*clone-child/ { 2021-04-04T09:34:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: format = 2 2021-04-04T09:34:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: } 2021-04-04T09:34:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: END { 2021-04-04T09:34:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: if (!parent) exit 1 2021-04-04T09:34:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: print format 2021-04-04T09:34:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: } 2021-04-04T09:34:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 1 = 1 2021-04-04T09:34:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog - clone v2 2021-04-04T09:34:18.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:34:18.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:34:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:34:18 - clone v2 2021-04-04T09:34:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:34:18 - clone v2 2021-04-04T09:34:18.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ parent_snap=snap_v2 2021-04-04T09:34:18.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap_v2 2021-04-04T09:34:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:18.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_v2 2021-04-04T09:34:18.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap_v2 2021-04-04T09:34:19.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-04-04T09:34:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clone_image cluster2 mirror_parent test_parent snap_v2 mirror test_clone_v2 --rbd-default-clone-format 2 2021-04-04T09:34:19.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:19.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local parent_pool=mirror_parent 2021-04-04T09:34:19.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local parent_image=test_parent 2021-04-04T09:34:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local parent_snap=snap_v2 2021-04-04T09:34:19.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local clone_pool=mirror 2021-04-04T09:34:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local clone_image=test_clone_v2 2021-04-04T09:34:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 6 2021-04-04T09:34:19.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 clone mirror_parent/test_parent@snap_v2 mirror/test_clone_v2 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 2 2021-04-04T09:34:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_clone_format cluster2 mirror test_clone_v2 2021-04-04T09:34:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:19.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v2 2021-04-04T09:34:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 info mirror/test_clone_v2 2021-04-04T09:34:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk BEGIN { 2021-04-04T09:34:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: format = 1 2021-04-04T09:34:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: } 2021-04-04T09:34:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: $1 == "parent:" { 2021-04-04T09:34:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: parent = $2 2021-04-04T09:34:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: } 2021-04-04T09:34:19.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: /op_features: .*clone-child/ { 2021-04-04T09:34:19.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: format = 2 2021-04-04T09:34:19.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: } 2021-04-04T09:34:19.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: END { 2021-04-04T09:34:19.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: if (!parent) exit 1 2021-04-04T09:34:19.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: print format 2021-04-04T09:34:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: } 2021-04-04T09:34:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 2 = 2 2021-04-04T09:34:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test_clone_v2 2021-04-04T09:34:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:19.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v2 2021-04-04T09:34:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test_clone_v2 started 2021-04-04T09:34:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v2 2021-04-04T09:34:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:34:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:34:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test_clone_v2 started 2021-04-04T09:34:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v2 2021-04-04T09:34:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:34:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:34:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:34:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:34:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local instance=cluster1 2021-04-04T09:34:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:34:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep+ -isleep 0 state 2021-04-04T09:34:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2021-04-04T09:34:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2021-04-04T09:34:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:20.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:34:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:34:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:34:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:34:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:34:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:20.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:34:20.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:20.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:20.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2021-04-04T09:34:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:20.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:34:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:34:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:20.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:34:20.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:34:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:34:20.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:34:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2021-04-04T09:34:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:20.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:34:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:20.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:34:20.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:20.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:34:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:34:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:34:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:20.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:34:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2021-04-04T09:34:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2021-04-04T09:34:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:34:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:34:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:34:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:34:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:34:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:34:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2021-04-04T09:34:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:34:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:34:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:34:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:34:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:34:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:22.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:34:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:22.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:22.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:22.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2021-04-04T09:34:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:34:22.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:34:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:22.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:22.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:34:22.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:34:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:34:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:34:22.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2021-04-04T09:34:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:34:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2021-04-04T09:34:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:34:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:34:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:34:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:34:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:24.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:34:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:24.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:34:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2021-04-04T09:34:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:34:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:34:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:24.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:24.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:34:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:34:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:34:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:34:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_clone_v2'] 2021-04-04T09:34:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:34:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:34:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:34:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:34:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:34:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:24.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:24.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:34:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_clone_v2 2021-04-04T09:34:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:34:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:34:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:34:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:34:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:34:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:34:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_clone_format cluster1 mirror test_clone_v2 2021-04-04T09:34:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v2 2021-04-04T09:34:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + rbd --cluster cluster1awk info BEGIN { 2021-04-04T09:34:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: format = 1 2021-04-04T09:34:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: } 2021-04-04T09:34:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: $1 == "parent:" { 2021-04-04T09:34:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: parent = $2 2021-04-04T09:34:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: } 2021-04-04T09:34:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: /op_features: .*clone-child/ { 2021-04-04T09:34:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: format = 2 2021-04-04T09:34:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: } 2021-04-04T09:34:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: END { 2021-04-04T09:34:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: if (!parent) exit 1 2021-04-04T09:34:24.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: print format 2021-04-04T09:34:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: } 2021-04-04T09:34:24.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: mirror/test_clone_v2 2021-04-04T09:34:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 2 = 2 2021-04-04T09:34:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap_v2 2021-04-04T09:34:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_v2 2021-04-04T09:34:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap_v2 2021-04-04T09:34:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2021-04-04T09:34:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2 2021-04-04T09:34:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap_v2 2021-04-04T09:34:25.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + grep -F trash (snap_v2) 2021-04-04T09:34:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd --cluster cluster2 snap ls mirror_parent/test_parent --all 2021-04-04T09:34:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 7 a225556f-b8c8-416f-8f58-b011b92dff42 128 MiB Sun Apr 4 09:34:19 2021 trash (snap_v2) 2021-04-04T09:34:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2021-04-04T09:34:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:34:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2021-04-04T09:34:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap_v2 2021-04-04T09:34:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2021-04-04T09:34:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -F trash (snap_v2) 2021-04-04T09:34:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ continue 2021-04-04T09:34:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:34:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2021-04-04T09:34:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:28.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap_v2 2021-04-04T09:34:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2021-04-04T09:34:28.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -F trash (snap_v2) 2021-04-04T09:34:28.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 7 e82de6e9-4641-46d4-9bcd-4e7421c82771 128 MiB Sun Apr 4 09:34:20 2021 trash (snap_v2) 2021-04-04T09:34:28.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_image_retry cluster2 mirror test_clone_v2 2021-04-04T09:34:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:28.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v2 2021-04-04T09:34:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:34:28.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_image cluster2 mirror test_clone_v2 2021-04-04T09:34:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v2 2021-04-04T09:34:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_clone_v2 2021-04-04T09:34:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 rm mirror/test_clone_v2 2021-04-04T09:34:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-04-04T09:34:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror test_clone_v2 deleted 2021-04-04T09:34:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v2 2021-04-04T09:34:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=deleted 2021-04-04T09:34:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:34:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:34:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:34:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test_clone_v2 2021-04-04T09:34:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v2 2021-04-04T09:34:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test_clone_v2 2021-04-04T09:34:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:34:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test_clone_v2: (2) No such file or directory 2021-04-04T09:34:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id= 2021-04-04T09:34:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:34:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test_clone_v2 deleted 2021-04-04T09:34:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v2 2021-04-04T09:34:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=deleted 2021-04-04T09:34:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:34:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:34:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:34:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test_clone_v2 2021-04-04T09:34:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v2 2021-04-04T09:34:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test_clone_v2 2021-04-04T09:34:29.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:34:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test_clone_v2: (2) No such file or directory 2021-04-04T09:34:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:34:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:34:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test deleted = deleted 2021-04-04T09:34:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_snap_removed_from_trash cluster2 mirror_parent test_parent snap_v2 2021-04-04T09:34:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2 2021-04-04T09:34:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap_v2 2021-04-04T09:34:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all 2021-04-04T09:34:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -F trash (snap_v2) 2021-04-04T09:34:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2021-04-04T09:34:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:34:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2021-04-04T09:34:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2021-04-04T09:34:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap_v2 2021-04-04T09:34:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2021-04-04T09:34:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -F trash (snap_v2) 2021-04-04T09:34:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:30.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog - clone v2 non-primary 2021-04-04T09:34:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:34:30.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:34:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:34:30 - clone v2 non-primary 2021-04-04T09:34:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:34:30 - clone v2 non-primary 2021-04-04T09:34:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snapshot cluster2 mirror_parent test_parent snap_v2 2021-04-04T09:34:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_v2 2021-04-04T09:34:30.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap create mirror_parent/test_parent@snap_v2 2021-04-04T09:34:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-04-04T09:34:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror_parent test_parent snap_v2 2021-04-04T09:34:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap_v2 2021-04-04T09:34:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:34:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror_parent/test_parent@snap_v2 2021-04-04T09:34:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error setting snapshot context: (2) No such file or directory 2021-04-04T09:34:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ continue 2021-04-04T09:34:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:34:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror_parent/test_parent@snap_v2 2021-04-04T09:34:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_parent': 2021-04-04T09:34:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:34:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:34:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2021-04-04T09:34:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ed3aaf7c64 2021-04-04T09:34:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ed3aaf7c64 2021-04-04T09:34:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:34:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:34:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:34:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:34:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:33:58 2021 2021-04-04T09:34:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:33:58 2021 2021-04-04T09:34:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:33:58 2021 2021-04-04T09:34:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2021-04-04T09:34:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10ed3aaf7c64 2021-04-04T09:34:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:34:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:34:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: ed3379f2-3a26-4e23-8dd8-e2567a8361ce 2021-04-04T09:34:34.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:34:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ clone_image cluster1 mirror_parent test_parent snap_v2 mirror test_clone_v2 --rbd-default-clone-format 2 2021-04-04T09:34:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:34.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local parent_pool=mirror_parent 2021-04-04T09:34:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local parent_image=test_parent 2021-04-04T09:34:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local parent_snap=snap_v2 2021-04-04T09:34:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local clone_pool=mirror 2021-04-04T09:34:34.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local clone_image=test_clone_v2 2021-04-04T09:34:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 6 2021-04-04T09:34:34.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 clone mirror_parent/test_parent@snap_v2 mirror/test_clone_v2 --image-feature layering,exclusive-lock,journaling --rbd-default-clone-format 2 2021-04-04T09:34:34.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror_parent test_parent snap_v2 2021-04-04T09:34:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_v2 2021-04-04T09:34:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror_parent/test_parent@snap_v2 2021-04-04T09:34:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2021-04-04T09:34:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_snap_removed_from_trash cluster2 mirror_parent test_parent snap_v2 2021-04-04T09:34:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_snap_moved_to_trash cluster2 mirror_parent test_parent snap_v2 2021-04-04T09:34:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap_v2 2021-04-04T09:34:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap ls mirror_parent/test_parent --all 2021-04-04T09:34:35.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -F trash (snap_v2) 2021-04-04T09:34:35.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2021-04-04T09:34:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:34:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2021-04-04T09:34:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:36.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap_v2 2021-04-04T09:34:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2021-04-04T09:34:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -F trash (snap_v2) 2021-04-04T09:34:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 10 867b344c-2d3b-4d80-a64b-f7f5c9d07fa0 128 MiB Sun Apr 4 09:34:33 2021 trash (snap_v2) 2021-04-04T09:34:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_image_retry cluster1 mirror test_clone_v2 2021-04-04T09:34:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v2 2021-04-04T09:34:36.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:34:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_image cluster1 mirror test_clone_v2 2021-04-04T09:34:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:36.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_clone_v2 2021-04-04T09:34:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 snap purge mirror/test_clone_v2 2021-04-04T09:34:36.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 rm mirror/test_clone_v2 2021-04-04T09:34:36.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-04-04T09:34:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2021-04-04T09:34:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:34:36.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_snap_removed_from_trash cluster1 mirror_parent test_parent snap_v2 2021-04-04T09:34:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_snap_moved_to_trash cluster1 mirror_parent test_parent snap_v2 2021-04-04T09:34:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror_parent 2021-04-04T09:34:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_parent 2021-04-04T09:34:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap_v2 2021-04-04T09:34:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 snap ls mirror_parent/test_parent --all 2021-04-04T09:34:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -F trash (snap_v2) 2021-04-04T09:34:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog TEST: data pool 2021-04-04T09:34:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:34:37.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:34:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:34:37 TEST: data pool 2021-04-04T09:34:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:34:37 TEST: data pool 2021-04-04T09:34:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ dp_image=test_data_pool 2021-04-04T09:34:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror test_data_pool 128 --data-pool mirror_parent 2021-04-04T09:34:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:37.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2021-04-04T09:34:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n 128 ] 2021-04-04T09:34:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128 2021-04-04T09:34:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool mirror_parent mirror/test_data_pool 2021-04-04T09:34:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_data_pool cluster2 mirror test_data_pool 2021-04-04T09:34:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 info mirror/test_data_pool 2021-04-04T09:34:37.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk $1 == "data_pool:" {print $2} 2021-04-04T09:34:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ data_pool=mirror_parent 2021-04-04T09:34:37.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test mirror_parent = mirror_parent 2021-04-04T09:34:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test_data_pool 2021-04-04T09:34:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:37.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test_data_pool started 2021-04-04T09:34:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:37.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:34:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:34:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test_data_pool started 2021-04-04T09:34:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:34:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:34:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:34:38.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test_data_pool 2021-04-04T09:34:38.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:34:38.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster=cluster1 2021-04-04T09:34:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1grep 2021-04-04T09:34:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: -i+ statelocal 2021-04-04T09:34:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: loop_instance 2021-04-04T09:34:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:34:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool 2021-04-04T09:34:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:39.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:39.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool 2021-04-04T09:34:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2021-04-04T09:34:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:39.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_data_pool 2021-04-04T09:34:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:34:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:34:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:34:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:34:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:34:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:34:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:39.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_data_pool 2021-04-04T09:34:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2021-04-04T09:34:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:39.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test_data_pool 2021-04-04T09:34:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:34:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:34:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:39.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:39.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:39.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:39.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:39.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:39.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:39.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:34:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_data_pool 2021-04-04T09:34:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2021-04-04T09:34:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:39.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test_data_pool 2021-04-04T09:34:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:34:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:34:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:39.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:39.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:34:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:34:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:34:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:39.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:39.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:34:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:39.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_data_pool 2021-04-04T09:34:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2021-04-04T09:34:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:40.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool 2021-04-04T09:34:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:40.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:40.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:40.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:40.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:40.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool 2021-04-04T09:34:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2021-04-04T09:34:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:40.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test_data_pool 2021-04-04T09:34:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:34:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:34:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:40.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:40.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:40.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:34:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:34:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:34:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:34:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:34:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test_data_pool 2021-04-04T09:34:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2021-04-04T09:34:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:40.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test_data_pool 2021-04-04T09:34:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:34:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:34:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:40.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:34:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:34:40.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:34:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:34:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:34:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:34:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:34:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:34:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:34:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test_data_pool 2021-04-04T09:34:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2021-04-04T09:34:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test_data_pool 2021-04-04T09:34:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:34:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:34:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:40.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:34:40.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:34:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:34:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:34:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:34:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:34:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:34:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:34:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:34:40.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test_data_pool 2021-04-04T09:34:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_data_pool'] 2021-04-04T09:34:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:34:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:34:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:34:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test_data_pool 2021-04-04T09:34:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:42.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test_data_pool 2021-04-04T09:34:43.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:34:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:34:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:34:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:34:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:34:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:34:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_data_pool cluster1 mirror test_data_pool 2021-04-04T09:34:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test_data_pool 2021-04-04T09:34:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk $1 == "data_pool:" {print $2} 2021-04-04T09:34:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ data_pool=mirror_parent 2021-04-04T09:34:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test mirror_parent = mirror_parent 2021-04-04T09:34:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snapshot cluster2 mirror test_data_pool snap1 2021-04-04T09:34:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap1 2021-04-04T09:34:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap create mirror/test_data_pool@snap1 2021-04-04T09:34:44.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-04-04T09:34:44.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test_data_pool 100 2021-04-04T09:34:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:34:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:34:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:34:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:34:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:34:44.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:34:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:34:44.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2500 bytes/sec: 9.8 MiB/s 2021-04-04T09:34:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snapshot cluster2 mirror test_data_pool snap2 2021-04-04T09:34:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap2 2021-04-04T09:34:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap create mirror/test_data_pool@snap2 2021-04-04T09:34:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-04-04T09:34:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test_data_pool 100 2021-04-04T09:34:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:45.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:34:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:34:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:34:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:34:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:34:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:34:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:34:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 531.915 bytes/sec: 2.1 MiB/s 2021-04-04T09:34:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_data_pool 2021-04-04T09:34:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:34:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:34:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test_data_pool 2021-04-04T09:34:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:34:45.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:34:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:34:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:34:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:34:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test_data_pool 2021-04-04T09:34:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:34:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:34:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test_data_pool 2021-04-04T09:34:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool 2021-04-04T09:34:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:34:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:34:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:34:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:34:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool 2021-04-04T09:34:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:45.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:45.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:45.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:45.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool 2021-04-04T09:34:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test_data_pool 2021-04-04T09:34:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test_data_pool 2021-04-04T09:34:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:34:45.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_data_pool.status 2021-04-04T09:34:45.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_data_pool.status 2021-04-04T09:34:45.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:45.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:45.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2021-04-04T09:34:45.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:34:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:34:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:34:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T09:34:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2]]], state=connected] 2021-04-04T09:34:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:45.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=5, entry_tid=101 2021-04-04T09:34:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test_data_pool 2021-04-04T09:34:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test_data_pool ..* 2021-04-04T09:34:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:34:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_data_pool.status 2021-04-04T09:34:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_data_pool.status 2021-04-04T09:34:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2021-04-04T09:34:45.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:34:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:34:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:34:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T09:34:46.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2]]], state=connected] 2021-04-04T09:34:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=1, tag_tid=2, entry_tid=1 2021-04-04T09:34:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=5, entry_tid=101 -a object_number=1, tag_tid=5, entry_tid=101 = object_number=1, tag_tid=2, entry_tid=1 2021-04-04T09:34:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=1, tag_tid=2, entry_tid=1 != 2021-04-04T09:34:46.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:34:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=1, tag_tid=2, entry_tid=1 = 2021-04-04T09:34:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_mirror_pos=object_number=1, tag_tid=2, entry_tid=1 2021-04-04T09:34:46.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=5, entry_tid=101 2021-04-04T09:34:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo tag_tid=[0-9]* 2021-04-04T09:34:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:34:46.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_tag=5 2021-04-04T09:34:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=2, entry_tid=1 2021-04-04T09:34:46.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo tag_tid=[0-9]* 2021-04-04T09:34:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:34:46.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_tag=2 2021-04-04T09:34:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=5, entry_tid=101 2021-04-04T09:34:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo entry_tid=[0-9]* 2021-04-04T09:34:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:34:46.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_entry=101 2021-04-04T09:34:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=2, entry_tid=1 2021-04-04T09:34:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo entry_tid=[0-9]* 2021-04-04T09:34:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:34:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_entry=1 2021-04-04T09:34:46.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 5 = 2 -a 101 -le 1 2021-04-04T09:34:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:34:46.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:34:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test_data_pool 2021-04-04T09:34:46.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:46.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:46.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:34:46.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:34:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test_data_pool 2021-04-04T09:34:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool 2021-04-04T09:34:46.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:34:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:34:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:34:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:34:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool 2021-04-04T09:34:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool 2021-04-04T09:34:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test_data_pool 2021-04-04T09:34:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test_data_pool 2021-04-04T09:34:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:34:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2021-04-04T09:34:46.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:34:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:34:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:34:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T09:34:46.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=101], [object_number=0, tag_tid=3, entry_tid=100], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected] 2021-04-04T09:34:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=5, entry_tid=101 2021-04-04T09:34:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test_data_pool 2021-04-04T09:34:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:46.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test_data_pool ..* 2021-04-04T09:34:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:34:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:46.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2021-04-04T09:34:46.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:34:46.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:34:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:34:46.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T09:34:46.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=101], [object_number=0, tag_tid=3, entry_tid=100], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected] 2021-04-04T09:34:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=1, tag_tid=3, entry_tid=101 2021-04-04T09:34:46.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=5, entry_tid=101 -a object_number=1, tag_tid=5, entry_tid=101 = object_number=1, tag_tid=3, entry_tid=101 2021-04-04T09:34:46.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=1, tag_tid=3, entry_tid=101 != object_number=1, tag_tid=2, entry_tid=1 2021-04-04T09:34:46.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:34:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=1, tag_tid=3, entry_tid=101 = object_number=1, tag_tid=2, entry_tid=1 2021-04-04T09:34:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_mirror_pos=object_number=1, tag_tid=3, entry_tid=101 2021-04-04T09:34:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=5, entry_tid=101 2021-04-04T09:34:46.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo tag_tid=[0-9]* 2021-04-04T09:34:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:34:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_tag=5 2021-04-04T09:34:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=3, entry_tid=101 2021-04-04T09:34:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + grep -Eocut tag_tid=[0-9]* -d= 2021-04-04T09:34:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: -f 2 2021-04-04T09:34:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_tag=3 2021-04-04T09:34:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=5, entry_tid=101 2021-04-04T09:34:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo entry_tid=[0-9]* 2021-04-04T09:34:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:34:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_entry=101 2021-04-04T09:34:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=3, entry_tid=101 2021-04-04T09:34:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo entry_tid=[0-9]* 2021-04-04T09:34:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:34:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_entry=101 2021-04-04T09:34:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 5 = 3 -a 101 -le 101 2021-04-04T09:34:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:34:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:34:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test_data_pool 2021-04-04T09:34:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:46.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:34:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:34:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test_data_pool 2021-04-04T09:34:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool 2021-04-04T09:34:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:34:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:46.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:34:46.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:34:46.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:34:46.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool 2021-04-04T09:34:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:46.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:46.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool 2021-04-04T09:34:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test_data_pool 2021-04-04T09:34:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test_data_pool 2021-04-04T09:34:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:34:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2021-04-04T09:34:46.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:34:46.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:34:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:34:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T09:34:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected] 2021-04-04T09:34:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=5, entry_tid=101 2021-04-04T09:34:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test_data_pool 2021-04-04T09:34:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test_data_pool ..* 2021-04-04T09:34:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:34:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2021-04-04T09:34:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:34:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:34:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:34:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T09:34:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=4, entry_tid=1], [object_number=0, tag_tid=4, entry_tid=0], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected] 2021-04-04T09:34:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=1, tag_tid=4, entry_tid=1 2021-04-04T09:34:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=5, entry_tid=101 -a object_number=1, tag_tid=5, entry_tid=101 = object_number=1, tag_tid=4, entry_tid=1 2021-04-04T09:34:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=1, tag_tid=4, entry_tid=1 != object_number=1, tag_tid=3, entry_tid=101 2021-04-04T09:34:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:34:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test object_number=1, tag_tid=4, entry_tid=1 = object_number=1, tag_tid=3, entry_tid=101 2021-04-04T09:34:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_mirror_pos=object_number=1, tag_tid=4, entry_tid=1 2021-04-04T09:34:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=5, entry_tid=101 2021-04-04T09:34:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + grep -Eo tag_tid=[0-9]* 2021-04-04T09:34:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:cut -d= -f 2 2021-04-04T09:34:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_tag=5 2021-04-04T09:34:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=4, entry_tid=1 2021-04-04T09:34:47.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo tag_tid=[0-9]* 2021-04-04T09:34:47.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:34:47.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_tag=4 2021-04-04T09:34:47.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=5, entry_tid=101 2021-04-04T09:34:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo entry_tid=[0-9]* 2021-04-04T09:34:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:34:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_entry=101 2021-04-04T09:34:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo object_number=1, tag_tid=4, entry_tid=1 2021-04-04T09:34:47.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -Eo entry_tid=[0-9]* 2021-04-04T09:34:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cut -d= -f 2 2021-04-04T09:34:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_entry=1 2021-04-04T09:34:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 5 = 4 -a 101 -le 1 2021-04-04T09:34:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:34:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:34:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test_data_pool 2021-04-04T09:34:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:47.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:34:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:34:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test_data_pool 2021-04-04T09:34:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_data_pool 2021-04-04T09:34:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:34:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:34:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:34:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:34:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:34:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:34:47.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_data_pool 2021-04-04T09:34:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:34:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:34:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:34:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:34:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:47.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:34:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:34:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:34:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:34:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:34:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:34:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:34:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:34:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:34:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:34:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:34:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:34:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:34:47.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:34:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:34:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_data_pool 2021-04-04T09:34:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test_data_pool 2021-04-04T09:34:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test_data_pool 2021-04-04T09:34:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:34:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_data_pool.status 2021-04-04T09:34:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + echo cluster2-mirror-test_data_pool.status 2021-04-04T09:34:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:sed -e s|[/ ]|_|g 2021-04-04T09:34:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2021-04-04T09:34:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:34:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:34:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:34:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T09:34:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T09:34:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=5, entry_tid=101 2021-04-04T09:34:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test_data_pool 2021-04-04T09:34:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test_data_pool ..* 2021-04-04T09:34:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:34:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_data_pool.status 2021-04-04T09:34:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_data_pool.status 2021-04-04T09:34:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test_data_pool 2021-04-04T09:34:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:47.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:34:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:34:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:34:47.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T09:34:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T09:34:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.status 2021-04-04T09:34:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=1, tag_tid=5, entry_tid=101 2021-04-04T09:34:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=5, entry_tid=101 -a object_number=1, tag_tid=5, entry_tid=101 = object_number=1, tag_tid=5, entry_tid=101 2021-04-04T09:34:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_data_pool up+replaying primary_position 2021-04-04T09:34:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:34:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:34:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:34:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test_data_pool up+replaying primary_position 2021-04-04T09:34:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:34:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:34:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:34:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test_data_pool.mirror_status 2021-04-04T09:34:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test_data_pool.mirror_status 2021-04-04T09:34:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool.mirror_status 2021-04-04T09:34:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test_data_pool 2021-04-04T09:34:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool.mirror_status 2021-04-04T09:34:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test_data_pool: 2021-04-04T09:34:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 63d2ca14-0fae-4993-96ee-cffbec8f3ffc 2021-04-04T09:34:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:34:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T09:34:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2021-04-04T09:34:48.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:34:41 2021-04-04T09:34:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:34:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:34:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:34:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:34:48.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:34:47 2021-04-04T09:34:48.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool.mirror_status 2021-04-04T09:34:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:34:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .*primary_position /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool.mirror_status 2021-04-04T09:34:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T09:34:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:34:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:34:48.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:34:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:34:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool.mirror_status 2021-04-04T09:34:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:34:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool.mirror_status 2021-04-04T09:34:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:34:41 2021-04-04T09:34:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_data_pool up+replaying 2021-04-04 09:34:41 2021-04-04T09:34:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:34:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:34:41 2021-04-04T09:34:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:34:48.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:34:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:34:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:34:48.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:34:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:34:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_data_pool']/last_update 2021-04-04T09:34:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:34:41 2021-04-04T09:34:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_data_pool']/state 2021-04-04T09:34:48.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:34:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:34:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:34:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:34:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_images mirror test_data_pool@snap1 2021-04-04T09:34:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool@snap1 2021-04-04T09:34:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2021-04-04T09:34:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_data_pool@snap1.export 2021-04-04T09:34:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_data_pool@snap1.export 2021-04-04T09:34:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool@snap1.export 2021-04-04T09:34:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test_data_pool@snap1.export 2021-04-04T09:34:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test_data_pool@snap1.export 2021-04-04T09:34:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool@snap1.export 2021-04-04T09:34:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool@snap1.export 2021-04-04T09:34:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool@snap1 /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool@snap1.export 2021-04-04T09:34:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:34:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool@snap1 /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool@snap1.export 2021-04-04T09:34:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:34:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool@snap1.export 2021-04-04T09:34:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool@snap1.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool@snap1.export 2021-04-04T09:34:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_images mirror test_data_pool@snap2 2021-04-04T09:34:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool@snap2 2021-04-04T09:34:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2021-04-04T09:34:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_data_pool@snap2.export 2021-04-04T09:34:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_data_pool@snap2.export 2021-04-04T09:34:49.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool@snap2.export 2021-04-04T09:34:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test_data_pool@snap2.export 2021-04-04T09:34:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test_data_pool@snap2.export 2021-04-04T09:34:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool@snap2.export 2021-04-04T09:34:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool@snap2.export 2021-04-04T09:34:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool@snap2 /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool@snap2.export 2021-04-04T09:34:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:34:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool@snap2 /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool@snap2.export 2021-04-04T09:34:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:34:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool@snap2.export 2021-04-04T09:34:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool@snap2.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool@snap2.export 2021-04-04T09:34:49.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_images mirror test_data_pool 2021-04-04T09:34:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:34:49.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2021-04-04T09:34:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_data_pool.export 2021-04-04T09:34:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_data_pool.export 2021-04-04T09:34:49.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.export 2021-04-04T09:34:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test_data_pool.export 2021-04-04T09:34:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test_data_pool.export 2021-04-04T09:34:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:34:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool.export 2021-04-04T09:34:49.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool.export 2021-04-04T09:34:49.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 export mirror/test_data_pool /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.export 2021-04-04T09:34:50.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:34:50.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 export mirror/test_data_pool /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool.export 2021-04-04T09:34:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:34:50.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool.export 2021-04-04T09:34:50.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_data_pool.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_data_pool.export 2021-04-04T09:34:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog TEST: disable mirroring / delete non-primary image 2021-04-04T09:34:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:34:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:34:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:34:50 TEST: disable mirroring / delete non-primary image 2021-04-04T09:34:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:34:50 TEST: disable mirroring / delete non-primary image 2021-04-04T09:34:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image2=test2 2021-04-04T09:34:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image3=test3 2021-04-04T09:34:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image4=test4 2021-04-04T09:34:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image5=test5 2021-04-04T09:34:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror test2 2021-04-04T09:34:50.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:34:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2021-04-04T09:34:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:34:50.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test2 2021-04-04T09:34:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test2 100 2021-04-04T09:34:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:34:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:34:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:34:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:34:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:34:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test2 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:34:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:34:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:34:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2083.33 bytes/sec: 8.1 MiB/s 2021-04-04T09:34:50.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snapshot cluster2 mirror test2 snap1 2021-04-04T09:34:50.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:50.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:34:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap1 2021-04-04T09:34:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap1 2021-04-04T09:34:51.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-04-04T09:34:51.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snapshot cluster2 mirror test2 snap2 2021-04-04T09:34:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:34:51.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap2 2021-04-04T09:34:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap2 2021-04-04T09:34:52.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-04-04T09:34:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ test2 = test4 ] 2021-04-04T09:34:52.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ test2 = test5 ] 2021-04-04T09:34:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test2 100 2021-04-04T09:34:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:52.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:34:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:34:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:34:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:34:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:34:52.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test2 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:34:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:34:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:34:52.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 543.478 bytes/sec: 2.1 MiB/s 2021-04-04T09:34:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror test2 present 2021-04-04T09:34:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:52.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:34:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=present 2021-04-04T09:34:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:34:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:34:52.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:34:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test2 2021-04-04T09:34:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:34:52.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test2 2021-04-04T09:34:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:34:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test2: (2) No such file or directory 2021-04-04T09:34:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id= 2021-04-04T09:34:52.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:34:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test2 present 2021-04-04T09:34:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:52.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:34:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:34:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:34:52.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:34:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:34:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test2 2021-04-04T09:34:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:52.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:34:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test2 2021-04-04T09:34:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:34:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id=10e044d10244 2021-04-04T09:34:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 10e044d10244 2021-04-04T09:34:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -z -o = 10e044d10244 2021-04-04T09:34:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=present 2021-04-04T09:34:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = present 2021-04-04T09:34:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test2 snap2 2021-04-04T09:34:52.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:34:52.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap2 2021-04-04T09:34:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:34:52.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:53.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2 2021-04-04T09:34:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error setting snapshot context: (2) No such file or directory 2021-04-04T09:34:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ continue 2021-04-04T09:34:53.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:34:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2 2021-04-04T09:34:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test2': 2021-04-04T09:34:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:34:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:34:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 3 2021-04-04T09:34:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e044d10244 2021-04-04T09:34:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e044d10244 2021-04-04T09:34:55.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:34:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:34:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:34:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:34:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:52 2021 2021-04-04T09:34:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:52 2021 2021-04-04T09:34:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:52 2021 2021-04-04T09:34:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2021-04-04T09:34:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10e044d10244 2021-04-04T09:34:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:34:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:34:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 09a6fbfb-f68f-4e71-98d3-0b0c50ff9d74 2021-04-04T09:34:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:34:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:55.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror test3 2021-04-04T09:34:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:55.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:55.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:34:55.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:34:55.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2021-04-04T09:34:55.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:34:55.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test3 2021-04-04T09:34:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test3 100 2021-04-04T09:34:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:34:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:34:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:34:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:34:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:34:55.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test3 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:34:55.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:34:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:34:55.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2083.33 bytes/sec: 8.1 MiB/s 2021-04-04T09:34:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snapshot cluster2 mirror test3 snap1 2021-04-04T09:34:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:34:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap1 2021-04-04T09:34:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap create mirror/test3@snap1 2021-04-04T09:34:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-04-04T09:34:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snapshot cluster2 mirror test3 snap2 2021-04-04T09:34:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:34:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap2 2021-04-04T09:34:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap create mirror/test3@snap2 2021-04-04T09:34:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-04-04T09:34:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ test3 = test4 ] 2021-04-04T09:34:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ test3 = test5 ] 2021-04-04T09:34:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test3 100 2021-04-04T09:34:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:34:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:34:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:34:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:34:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:34:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:34:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test3 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:34:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:34:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:34:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 609.756 bytes/sec: 2.4 MiB/s 2021-04-04T09:34:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror test3 present 2021-04-04T09:34:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:34:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=present 2021-04-04T09:34:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:34:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:34:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:34:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test3 2021-04-04T09:34:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:34:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test3 2021-04-04T09:34:57.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:34:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test3: (2) No such file or directory 2021-04-04T09:34:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id= 2021-04-04T09:34:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:34:57.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test3 present 2021-04-04T09:34:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:34:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:34:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:34:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:34:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:34:57.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test3 2021-04-04T09:34:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:34:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test3 2021-04-04T09:34:57.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:34:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test3: (2) No such file or directory 2021-04-04T09:34:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:34:57.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:34:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = deleted 2021-04-04T09:34:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test3 present 2021-04-04T09:34:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:34:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:34:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:34:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:34:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:34:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test3 2021-04-04T09:34:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:34:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test3 2021-04-04T09:34:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:34:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id=10e7c7fffde7 2021-04-04T09:34:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 10e7c7fffde7 2021-04-04T09:34:58.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -z -o = 10e7c7fffde7 2021-04-04T09:34:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=present 2021-04-04T09:34:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = present 2021-04-04T09:34:58.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:34:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test3 snap2 2021-04-04T09:34:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:34:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:34:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:34:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap2 2021-04-04T09:34:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:34:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:34:59.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test3@snap2 2021-04-04T09:34:59.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error setting snapshot context: (2) No such file or directory 2021-04-04T09:34:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ continue 2021-04-04T09:34:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:35:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test3@snap2 2021-04-04T09:35:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test3': 2021-04-04T09:35:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:35:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:35:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 3 2021-04-04T09:35:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e7c7fffde7 2021-04-04T09:35:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e7c7fffde7 2021-04-04T09:35:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:35:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:35:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:35:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:35:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:57 2021 2021-04-04T09:35:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:57 2021 2021-04-04T09:35:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:57 2021 2021-04-04T09:35:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2021-04-04T09:35:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10e7c7fffde7 2021-04-04T09:35:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:35:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:35:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 409477f3-cbd2-4eff-9e17-c87e509f4e40 2021-04-04T09:35:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:35:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror test4 2021-04-04T09:35:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2021-04-04T09:35:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:35:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test4 2021-04-04T09:35:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test4 100 2021-04-04T09:35:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:35:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:35:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:35:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test4 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:35:01.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:35:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:35:01.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2500 bytes/sec: 9.8 MiB/s 2021-04-04T09:35:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snapshot cluster2 mirror test4 snap1 2021-04-04T09:35:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap1 2021-04-04T09:35:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap create mirror/test4@snap1 2021-04-04T09:35:02.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-04-04T09:35:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snapshot cluster2 mirror test4 snap2 2021-04-04T09:35:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap2 2021-04-04T09:35:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap create mirror/test4@snap2 2021-04-04T09:35:03.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-04-04T09:35:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ test4 = test4 ] 2021-04-04T09:35:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ protect_snapshot cluster2 mirror test4 snap1 2021-04-04T09:35:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap1 2021-04-04T09:35:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap protect mirror/test4@snap1 2021-04-04T09:35:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ protect_snapshot cluster2 mirror test4 snap2 2021-04-04T09:35:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap2 2021-04-04T09:35:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap protect mirror/test4@snap2 2021-04-04T09:35:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test4 100 2021-04-04T09:35:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:35:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:35:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:35:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test4 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:35:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:35:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:35:03.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 625 bytes/sec: 2.4 MiB/s 2021-04-04T09:35:03.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror test4 present 2021-04-04T09:35:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=present 2021-04-04T09:35:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:35:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:03.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test4 2021-04-04T09:35:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test4 2021-04-04T09:35:03.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test4: (2) No such file or directory 2021-04-04T09:35:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id= 2021-04-04T09:35:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:35:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test4 present 2021-04-04T09:35:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:35:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:03.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:35:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:35:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test4 2021-04-04T09:35:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test4 2021-04-04T09:35:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test4: (2) No such file or directory 2021-04-04T09:35:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:35:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = deleted 2021-04-04T09:35:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:35:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test4 present 2021-04-04T09:35:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:35:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:04.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:35:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:35:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test4 2021-04-04T09:35:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:04.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test4 2021-04-04T09:35:04.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id=10eae37727c 2021-04-04T09:35:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 10eae37727c 2021-04-04T09:35:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -z -o = 10eae37727c 2021-04-04T09:35:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=present 2021-04-04T09:35:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = present 2021-04-04T09:35:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test4 snap2 2021-04-04T09:35:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap2 2021-04-04T09:35:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:35:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:35:05.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test4@snap2 2021-04-04T09:35:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test4': 2021-04-04T09:35:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:35:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:35:05.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2021-04-04T09:35:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10eae37727c 2021-04-04T09:35:05.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10eae37727c 2021-04-04T09:35:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:35:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:35:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:35:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:35:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:35:03 2021 2021-04-04T09:35:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:35:03 2021 2021-04-04T09:35:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:35:03 2021 2021-04-04T09:35:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2021-04-04T09:35:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10eae37727c 2021-04-04T09:35:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:35:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:35:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 87727dd3-4985-41eb-bd8e-3591e277f847 2021-04-04T09:35:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:35:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror test5 2021-04-04T09:35:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2021-04-04T09:35:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:35:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/test5 2021-04-04T09:35:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test5 100 2021-04-04T09:35:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:35:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:35:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:35:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test5 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:35:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:35:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:35:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2500 bytes/sec: 9.8 MiB/s 2021-04-04T09:35:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snapshot cluster2 mirror test5 snap1 2021-04-04T09:35:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap1 2021-04-04T09:35:06.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap create mirror/test5@snap1 2021-04-04T09:35:06.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-04-04T09:35:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snapshot cluster2 mirror test5 snap2 2021-04-04T09:35:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:06.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap2 2021-04-04T09:35:06.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap create mirror/test5@snap2 2021-04-04T09:35:07.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-04-04T09:35:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ test5 = test4 ] 2021-04-04T09:35:07.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ test5 = test5 ] 2021-04-04T09:35:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ protect_snapshot cluster2 mirror test5 snap1 2021-04-04T09:35:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:07.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap1 2021-04-04T09:35:07.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap protect mirror/test5@snap1 2021-04-04T09:35:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ protect_snapshot cluster2 mirror test5 snap2 2021-04-04T09:35:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap2 2021-04-04T09:35:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap protect mirror/test5@snap2 2021-04-04T09:35:07.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test5 100 2021-04-04T09:35:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:07.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:35:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:35:07.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:35:07.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test5 --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:35:07.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:35:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:35:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 510.204 bytes/sec: 2.0 MiB/s 2021-04-04T09:35:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror test5 present 2021-04-04T09:35:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=present 2021-04-04T09:35:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:35:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test5 2021-04-04T09:35:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test5 2021-04-04T09:35:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test5: (2) No such file or directory 2021-04-04T09:35:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id= 2021-04-04T09:35:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:35:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test5 present 2021-04-04T09:35:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:35:07.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:35:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:35:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test5 2021-04-04T09:35:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test5 2021-04-04T09:35:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test5: (2) No such file or directory 2021-04-04T09:35:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:35:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = deleted 2021-04-04T09:35:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:35:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test5 present 2021-04-04T09:35:08.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:35:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:35:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:35:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test5 2021-04-04T09:35:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:08.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test5 2021-04-04T09:35:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:09.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id=10dd4aaf04c8 2021-04-04T09:35:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 10dd4aaf04c8 2021-04-04T09:35:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -z -o = 10dd4aaf04c8 2021-04-04T09:35:09.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=present 2021-04-04T09:35:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = present 2021-04-04T09:35:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test5 snap2 2021-04-04T09:35:09.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap2 2021-04-04T09:35:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:35:09.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:35:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test5@snap2 2021-04-04T09:35:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test5': 2021-04-04T09:35:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:35:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:35:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2021-04-04T09:35:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10dd4aaf04c8 2021-04-04T09:35:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10dd4aaf04c8 2021-04-04T09:35:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:35:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:35:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:35:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:35:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:35:08 2021 2021-04-04T09:35:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:35:08 2021 2021-04-04T09:35:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:35:08 2021 2021-04-04T09:35:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2021-04-04T09:35:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10dd4aaf04c8 2021-04-04T09:35:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:35:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:35:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 690ce498-199d-4023-9c64-ff4d7599e591 2021-04-04T09:35:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:35:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_pool_mirror_mode cluster2 mirror image 2021-04-04T09:35:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=image 2021-04-04T09:35:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2021-04-04T09:35:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:note: changing mirroring mode from pool to image 2021-04-04T09:35:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ disable_mirror cluster2 mirror test2 2021-04-04T09:35:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test2 2021-04-04T09:35:10.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring disabled 2021-04-04T09:35:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ disable_mirror cluster2 mirror test4 2021-04-04T09:35:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test4 2021-04-04T09:35:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring disabled 2021-04-04T09:35:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ unprotect_snapshot cluster2 mirror test5 snap1 2021-04-04T09:35:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap1 2021-04-04T09:35:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test5@snap1 2021-04-04T09:35:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ unprotect_snapshot cluster2 mirror test5 snap2 2021-04-04T09:35:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap2 2021-04-04T09:35:10.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap unprotect mirror/test5@snap2 2021-04-04T09:35:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test3 snap1 2021-04-04T09:35:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:35:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap1 2021-04-04T09:35:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test3@snap1 2021-04-04T09:35:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2021-04-04T09:35:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test3 snap2 2021-04-04T09:35:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:11.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:35:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap2 2021-04-04T09:35:11.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test3@snap2 2021-04-04T09:35:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2021-04-04T09:35:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_image_retry cluster2 mirror test3 2021-04-04T09:35:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:35:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:35:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_image cluster2 mirror test3 2021-04-04T09:35:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:35:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 snap purge mirror/test3 2021-04-04T09:35:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 rm mirror/test3 2021-04-04T09:35:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-04-04T09:35:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test5 snap1 2021-04-04T09:35:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap1 2021-04-04T09:35:12.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test5@snap1 2021-04-04T09:35:14.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2021-04-04T09:35:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_snapshot cluster2 mirror test5 snap2 2021-04-04T09:35:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap2 2021-04-04T09:35:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rm mirror/test5@snap2 2021-04-04T09:35:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing snap: 100% complete...done. 2021-04-04T09:35:15.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_image_retry cluster2 mirror test5 2021-04-04T09:35:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:35:15.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_image cluster2 mirror test5 2021-04-04T09:35:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:15.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 snap purge mirror/test5 2021-04-04T09:35:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 rm mirror/test5 2021-04-04T09:35:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-04-04T09:35:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror test2 deleted 2021-04-04T09:35:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=deleted 2021-04-04T09:35:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:35:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test2 2021-04-04T09:35:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test2 2021-04-04T09:35:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id=10e044d10244 2021-04-04T09:35:15.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:35:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test2 deleted 10e044d10244 2021-04-04T09:35:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=deleted 2021-04-04T09:35:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id=10e044d10244 2021-04-04T09:35:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:35:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:35:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test2 2021-04-04T09:35:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test2 2021-04-04T09:35:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:15.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test2: (2) No such file or directory 2021-04-04T09:35:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:35:15.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test deleted = deleted 2021-04-04T09:35:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror test3 deleted 2021-04-04T09:35:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:15.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:35:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=deleted 2021-04-04T09:35:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:35:15.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test3 2021-04-04T09:35:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:35:15.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test3 2021-04-04T09:35:15.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:15.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test3: (2) No such file or directory 2021-04-04T09:35:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id= 2021-04-04T09:35:15.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:35:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test3 deleted 2021-04-04T09:35:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:35:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=deleted 2021-04-04T09:35:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:35:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:35:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test3 2021-04-04T09:35:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test3 2021-04-04T09:35:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test3 2021-04-04T09:35:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:15.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test3: (2) No such file or directory 2021-04-04T09:35:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:35:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:15.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test deleted = deleted 2021-04-04T09:35:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror test4 deleted 2021-04-04T09:35:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=deleted 2021-04-04T09:35:15.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:35:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test4 2021-04-04T09:35:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:15.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test4 2021-04-04T09:35:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:15.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test4: (2) No such file or directory 2021-04-04T09:35:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id= 2021-04-04T09:35:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:35:15.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test4 deleted 2021-04-04T09:35:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=deleted 2021-04-04T09:35:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:35:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:35:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test4 2021-04-04T09:35:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test4 2021-04-04T09:35:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test4: (2) No such file or directory 2021-04-04T09:35:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:35:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test deleted = deleted 2021-04-04T09:35:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror test5 deleted 2021-04-04T09:35:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=deleted 2021-04-04T09:35:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:35:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test5 2021-04-04T09:35:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test5 2021-04-04T09:35:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id=10dd4aaf04c8 2021-04-04T09:35:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:35:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test5 deleted 10dd4aaf04c8 2021-04-04T09:35:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=deleted 2021-04-04T09:35:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id=10dd4aaf04c8 2021-04-04T09:35:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:35:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:35:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test5 2021-04-04T09:35:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test5 2021-04-04T09:35:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id=10dd4aaf04c8 2021-04-04T09:35:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 10dd4aaf04c8 2021-04-04T09:35:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -z 10dd4aaf04c8 -o 10dd4aaf04c8 = 10dd4aaf04c8 2021-04-04T09:35:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=present 2021-04-04T09:35:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test deleted = present 2021-04-04T09:35:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:35:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test5 deleted 10dd4aaf04c8 2021-04-04T09:35:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=deleted 2021-04-04T09:35:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id=10dd4aaf04c8 2021-04-04T09:35:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:35:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:35:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test5 2021-04-04T09:35:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:17.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test5 2021-04-04T09:35:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test5 2021-04-04T09:35:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:17.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test5: (2) No such file or directory 2021-04-04T09:35:17.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:35:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test deleted = deleted 2021-04-04T09:35:17.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_pool_mirror_mode cluster2 mirror pool 2021-04-04T09:35:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:17.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=pool 2021-04-04T09:35:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror pool 2021-04-04T09:35:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:note: changing mirroring mode from image to pool 2021-04-04T09:35:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_journaling cluster2 mirror test2 2021-04-04T09:35:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 feature enable mirror/test2 journaling 2021-04-04T09:35:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror test2 present 2021-04-04T09:35:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=present 2021-04-04T09:35:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:35:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test2 2021-04-04T09:35:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test2 2021-04-04T09:35:17.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test2: (2) No such file or directory 2021-04-04T09:35:17.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id= 2021-04-04T09:35:17.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:35:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test2 present 2021-04-04T09:35:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:35:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:17.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:35:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:35:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test2 2021-04-04T09:35:17.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test2 2021-04-04T09:35:17.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test2: (2) No such file or directory 2021-04-04T09:35:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:35:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = deleted 2021-04-04T09:35:17.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:35:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test2 present 2021-04-04T09:35:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:35:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:35:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:35:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test2 2021-04-04T09:35:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test2 2021-04-04T09:35:18.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test2: (2) No such file or directory 2021-04-04T09:35:18.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:35:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = deleted 2021-04-04T09:35:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:35:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test2 present 2021-04-04T09:35:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:35:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:35:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:35:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test2 2021-04-04T09:35:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test2 2021-04-04T09:35:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:20.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test2: (2) No such file or directory 2021-04-04T09:35:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:35:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = deleted 2021-04-04T09:35:20.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 4 2021-04-04T09:35:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test2 present 2021-04-04T09:35:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:35:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:35:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:35:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test2 2021-04-04T09:35:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:24.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test2 2021-04-04T09:35:24.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id=10e0b93cfe9e 2021-04-04T09:35:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 10e0b93cfe9e 2021-04-04T09:35:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -z -o = 10e0b93cfe9e 2021-04-04T09:35:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=present 2021-04-04T09:35:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = present 2021-04-04T09:35:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test2 snap2 2021-04-04T09:35:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap2 2021-04-04T09:35:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:35:24.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:35:25.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test2@snap2 2021-04-04T09:35:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test2': 2021-04-04T09:35:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:35:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:35:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2021-04-04T09:35:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0b93cfe9e 2021-04-04T09:35:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0b93cfe9e 2021-04-04T09:35:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:35:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:35:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:35:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:35:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:35:24 2021 2021-04-04T09:35:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:35:24 2021 2021-04-04T09:35:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:35:24 2021 2021-04-04T09:35:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2021-04-04T09:35:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10e0b93cfe9e 2021-04-04T09:35:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:35:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:35:25.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 80ed7469-0640-4a03-9952-a34e4d433832 2021-04-04T09:35:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:35:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test2 2021-04-04T09:35:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:25.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test2 started 2021-04-04T09:35:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:25.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:35:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:35:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:35:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test2 started 2021-04-04T09:35:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:35:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:35:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:35:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test2 2021-04-04T09:35:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:35:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster=cluster1 2021-04-04T09:35:26.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:35:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instancegrep 2021-04-04T09:35:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: -i state 2021-04-04T09:35:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:35:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:35:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:35:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test2 2021-04-04T09:35:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:35:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:26.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:35:26.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:26.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:35:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:35:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:35:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:26.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:26.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:35:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test2 2021-04-04T09:35:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test2'] 2021-04-04T09:35:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:35:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test2 2021-04-04T09:35:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:35:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:35:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:35:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:35:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:35:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:35:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test2 2021-04-04T09:35:27.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test2'] 2021-04-04T09:35:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:35:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test2 2021-04-04T09:35:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:35:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:35:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:35:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:35:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:35:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:35:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:35:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:35:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:35:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:35:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:35:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:35:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test2 2021-04-04T09:35:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test2'] 2021-04-04T09:35:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:35:27.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test2 2021-04-04T09:35:27.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:35:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:35:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:35:27.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:35:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:35:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:35:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:35:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:35:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:35:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:35:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:35:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:35:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test2 2021-04-04T09:35:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test2'] 2021-04-04T09:35:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:35:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:27.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:35:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:35:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:35:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test2 2021-04-04T09:35:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:35:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:35:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:35:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:35:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:35:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:28.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:28.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:35:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:28.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test2 2021-04-04T09:35:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test2'] 2021-04-04T09:35:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:35:28.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test2 2021-04-04T09:35:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:35:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:35:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:28.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:28.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:35:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:35:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:35:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:35:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:28.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test2 2021-04-04T09:35:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:35:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:35:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:35:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:35:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:35:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:35:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test2 2021-04-04T09:35:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:35:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:35:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test2 2021-04-04T09:35:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:35:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:35:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:35:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:35:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:35:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test2 2021-04-04T09:35:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:35:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test2 ] 2021-04-04T09:35:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test2 2021-04-04T09:35:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test2 2021-04-04T09:35:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:35:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:35:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:35:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:35:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:35:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:35:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test2 2021-04-04T09:35:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:35:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:35:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:35:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:35:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:35:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:35:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test2 2021-04-04T09:35:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test2'] 2021-04-04T09:35:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test2 2021-04-04T09:35:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:35:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:35:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:35:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:28.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:35:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test2 2021-04-04T09:35:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test2 2021-04-04T09:35:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test2 2021-04-04T09:35:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:35:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test2.status 2021-04-04T09:35:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test2.status 2021-04-04T09:35:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:35:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test2.status 2021-04-04T09:35:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test2.status 2021-04-04T09:35:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test2 2021-04-04T09:35:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:35:29.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:35:29.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:35:29.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T09:35:29.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T09:35:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test2.status 2021-04-04T09:35:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=3, tag_tid=1, entry_tid=3 2021-04-04T09:35:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test2 2021-04-04T09:35:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test2 ..* 2021-04-04T09:35:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:29.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:35:29.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test2.status 2021-04-04T09:35:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test2.status 2021-04-04T09:35:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:35:29.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test2.status 2021-04-04T09:35:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test2 2021-04-04T09:35:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test2.status 2021-04-04T09:35:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:35:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:35:29.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:35:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T09:35:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T09:35:29.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test2.status 2021-04-04T09:35:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=3, tag_tid=1, entry_tid=3 2021-04-04T09:35:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=3, tag_tid=1, entry_tid=3 -a object_number=3, tag_tid=1, entry_tid=3 = object_number=3, tag_tid=1, entry_tid=3 2021-04-04T09:35:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_images mirror test2 2021-04-04T09:35:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:35:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2021-04-04T09:35:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test2.export 2021-04-04T09:35:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test2.export 2021-04-04T09:35:29.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:35:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test2.export 2021-04-04T09:35:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test2.export 2021-04-04T09:35:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test2.export 2021-04-04T09:35:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:35:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test2.export 2021-04-04T09:35:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test2.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test2.export 2021-04-04T09:35:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 export mirror/test2 /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test2.export 2021-04-04T09:35:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:35:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 export mirror/test2 /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test2.export 2021-04-04T09:35:29.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:35:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test2.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test2.export 2021-04-04T09:35:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test2.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test2.export 2021-04-04T09:35:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_journaling cluster2 mirror test4 2021-04-04T09:35:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:29.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 feature enable mirror/test4 journaling 2021-04-04T09:35:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror test4 present 2021-04-04T09:35:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=present 2021-04-04T09:35:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:35:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test4 2021-04-04T09:35:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test4 2021-04-04T09:35:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test4: (2) No such file or directory 2021-04-04T09:35:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id= 2021-04-04T09:35:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:35:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test4 present 2021-04-04T09:35:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:35:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:35:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:35:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test4 2021-04-04T09:35:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:29.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:29.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test4 2021-04-04T09:35:29.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:29.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test4: (2) No such file or directory 2021-04-04T09:35:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:35:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = deleted 2021-04-04T09:35:29.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:35:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test4 present 2021-04-04T09:35:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:30.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:35:30.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:30.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:35:30.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:35:30.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test4 2021-04-04T09:35:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test4 2021-04-04T09:35:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:30.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test4: (2) No such file or directory 2021-04-04T09:35:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:35:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = deleted 2021-04-04T09:35:30.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:35:32.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test4 present 2021-04-04T09:35:32.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:35:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:35:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:35:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:35:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test4 2021-04-04T09:35:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test4 2021-04-04T09:35:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:35:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id=10e78fad30ae 2021-04-04T09:35:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 10e78fad30ae 2021-04-04T09:35:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -z -o = 10e78fad30ae 2021-04-04T09:35:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=present 2021-04-04T09:35:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = present 2021-04-04T09:35:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:33.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test4 snap2 2021-04-04T09:35:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap2 2021-04-04T09:35:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:35:33.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:35:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test4@snap2 2021-04-04T09:35:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test4': 2021-04-04T09:35:34.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:35:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:35:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2021-04-04T09:35:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e78fad30ae 2021-04-04T09:35:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e78fad30ae 2021-04-04T09:35:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:35:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:35:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:35:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:35:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:35:31 2021 2021-04-04T09:35:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:35:31 2021 2021-04-04T09:35:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:35:31 2021 2021-04-04T09:35:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2021-04-04T09:35:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10e78fad30ae 2021-04-04T09:35:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:35:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:35:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 698bca8d-5596-41e0-b6a2-71fc39245e79 2021-04-04T09:35:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:35:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test4 2021-04-04T09:35:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test4 started 2021-04-04T09:35:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:35:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:35:34.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:35:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test4 started 2021-04-04T09:35:35.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:35.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:35.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:35.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:35:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:35:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:35:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test4 2021-04-04T09:35:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:35:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:35:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:35:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:35:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:35:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:35:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:35:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test4 2021-04-04T09:35:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:35:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:35:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:35.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:35:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:35:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:35:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:35.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:35:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:35.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test4 2021-04-04T09:35:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test4'] 2021-04-04T09:35:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:35:35.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test4 2021-04-04T09:35:35.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:35:35.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:35.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:35:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:35.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:35.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:35:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:35:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:35:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:35.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:35:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test4 2021-04-04T09:35:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test4'] 2021-04-04T09:35:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:35:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test4 2021-04-04T09:35:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:35:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:35:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:35:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:35:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:35.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:35:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:35:35.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:35:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:35:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:35:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:35:35.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:35:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:35:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:35.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test4 2021-04-04T09:35:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test4'] 2021-04-04T09:35:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:35:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test4 2021-04-04T09:35:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:35:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:35.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:35:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:35:35.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:35:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:35:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:35:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:35:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:35:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:35:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:35:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:35:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:35:35.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test4 2021-04-04T09:35:35.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test4'] 2021-04-04T09:35:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:35:35.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:35.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:35:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:35:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:35:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test4 2021-04-04T09:35:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:35:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:35:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:36.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:35:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:35:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:36.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:35:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:36.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:35:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:36.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test4 2021-04-04T09:35:36.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test4'] 2021-04-04T09:35:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:35:36.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test4 2021-04-04T09:35:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:35:36.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:35:36.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:36.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:36.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:35:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:35:36.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:35:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:35:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:36.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test4 2021-04-04T09:35:36.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test4'] 2021-04-04T09:35:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:35:36.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test4 2021-04-04T09:35:36.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:35:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:35:36.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:35:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:36.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:35:36.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:35:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:35:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:35:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:36.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:35:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:35:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:36.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:35:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:36.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:35:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:35:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test4 2021-04-04T09:35:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:35:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + echo "state": "Replaying" 2021-04-04T09:35:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i Replaying 2021-04-04T09:35:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:35:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:35:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:35:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test4 2021-04-04T09:35:37.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:35:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:35:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test4 2021-04-04T09:35:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:35:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:37.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:35:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:35:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:35:37.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:35:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test4 2021-04-04T09:35:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:35:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test4 ] 2021-04-04T09:35:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test4 2021-04-04T09:35:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test4 2021-04-04T09:35:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:35:37.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:37.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:37.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:35:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:35:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:35:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:35:37.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:35:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test4 2021-04-04T09:35:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:35:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:35:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:37.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:35:37.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:35:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:35:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:37.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:35:37.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:37.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test4 2021-04-04T09:35:37.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test4'] 2021-04-04T09:35:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:37.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test4 2021-04-04T09:35:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:35:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:35:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:37.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:35:37.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:35:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:37.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:35:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:37.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:37.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:35:37.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:37.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:37.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test4 2021-04-04T09:35:37.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test4'] 2021-04-04T09:35:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test4 2021-04-04T09:35:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:35:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:37.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:35:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:35:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:37.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:35:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:37.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:35:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:35:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:35:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:35:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:37.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:35:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:35:37.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:37.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:37.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test4 2021-04-04T09:35:37.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test4 2021-04-04T09:35:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test4 2021-04-04T09:35:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:35:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test4.status 2021-04-04T09:35:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test4.status 2021-04-04T09:35:37.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:35:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test4.status 2021-04-04T09:35:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test4 2021-04-04T09:35:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test4.status 2021-04-04T09:35:37.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:35:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:35:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:35:37.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T09:35:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T09:35:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test4.status 2021-04-04T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=3, tag_tid=1, entry_tid=3 2021-04-04T09:35:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test4 2021-04-04T09:35:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test4 ..* 2021-04-04T09:35:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:35:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test4.status 2021-04-04T09:35:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test4.status 2021-04-04T09:35:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:35:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test4.status 2021-04-04T09:35:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test4 2021-04-04T09:35:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test4.status 2021-04-04T09:35:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:35:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:35:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:35:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T09:35:37.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T09:35:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test4.status 2021-04-04T09:35:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=3, tag_tid=1, entry_tid=3 2021-04-04T09:35:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=3, tag_tid=1, entry_tid=3 -a object_number=3, tag_tid=1, entry_tid=3 = object_number=3, tag_tid=1, entry_tid=3 2021-04-04T09:35:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_images mirror test4 2021-04-04T09:35:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test4 2021-04-04T09:35:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2021-04-04T09:35:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test4.export 2021-04-04T09:35:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test4.export 2021-04-04T09:35:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:35:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test4.export 2021-04-04T09:35:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test4.export 2021-04-04T09:35:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test4.export 2021-04-04T09:35:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:35:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test4.export 2021-04-04T09:35:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test4.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test4.export 2021-04-04T09:35:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 export mirror/test4 /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test4.export 2021-04-04T09:35:37.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:35:37.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 export mirror/test4 /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test4.export 2021-04-04T09:35:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:35:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test4.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test4.export 2021-04-04T09:35:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test4.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test4.export 2021-04-04T09:35:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog TEST: remove mirroring pool 2021-04-04T09:35:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + date +%F %T 2021-04-04T09:35:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:35:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:35:38 TEST: remove mirroring pool 2021-04-04T09:35:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:35:38 TEST: remove mirroring pool 2021-04-04T09:35:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pool=pool_to_remove 2021-04-04T09:35:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool create pool_to_remove 16 16 2021-04-04T09:35:39.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:pool 'pool_to_remove' already exists 2021-04-04T09:35:39.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 pool init pool_to_remove 2021-04-04T09:35:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool enable pool_to_remove pool 2021-04-04T09:35:42.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool create pool_to_remove 16 16 2021-04-04T09:35:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:pool 'pool_to_remove' already exists 2021-04-04T09:35:43.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 pool init pool_to_remove 2021-04-04T09:35:46.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool enable pool_to_remove pool 2021-04-04T09:35:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_add cluster1 pool_to_remove cluster2 2021-04-04T09:35:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=pool_to_remove 2021-04-04T09:35:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local client_cluster=cluster2 2021-04-04T09:35:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local remote_cluster=cluster2 2021-04-04T09:35:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local uuid_var_name 2021-04-04T09:35:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:35:46.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local error_code 2021-04-04T09:35:46.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local peer_uuid 2021-04-04T09:35:46.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2021-04-04T09:35:46.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool peer add pool_to_remove cluster2 2021-04-04T09:35:46.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_uuid=e87ff811-cd71-4341-9c7e-97d241f1c043 2021-04-04T09:35:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=0 2021-04-04T09:35:46.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2021-04-04T09:35:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 -eq 17 ] 2021-04-04T09:35:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 0 -eq 0 2021-04-04T09:35:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:35:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_add cluster2 pool_to_remove cluster1 2021-04-04T09:35:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=pool_to_remove 2021-04-04T09:35:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local client_cluster=cluster1 2021-04-04T09:35:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local remote_cluster=cluster1 2021-04-04T09:35:46.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local uuid_var_name 2021-04-04T09:35:46.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:35:46.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local error_code 2021-04-04T09:35:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local peer_uuid 2021-04-04T09:35:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2021-04-04T09:35:46.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 mirror pool peer add pool_to_remove cluster1 2021-04-04T09:35:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ peer_uuid=f2b9ad21-f037-4467-983d-91502f277319 2021-04-04T09:35:46.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ error_code=0 2021-04-04T09:35:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set -e 2021-04-04T09:35:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 -eq 17 ] 2021-04-04T09:35:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test 0 -eq 0 2021-04-04T09:35:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:35:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:35:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rdp_image=test_remove_data_pool 2021-04-04T09:35:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 pool_to_remove test 128 2021-04-04T09:35:46.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=pool_to_remove 2021-04-04T09:35:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:35:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2021-04-04T09:35:46.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n 128 ] 2021-04-04T09:35:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128 2021-04-04T09:35:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling pool_to_remove/test 2021-04-04T09:35:46.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror test_remove_data_pool 128 --data-pool pool_to_remove 2021-04-04T09:35:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:35:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2021-04-04T09:35:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n 128 ] 2021-04-04T09:35:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128 2021-04-04T09:35:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool pool_to_remove mirror/test_remove_data_pool 2021-04-04T09:35:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 pool_to_remove test 100 2021-04-04T09:35:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=pool_to_remove 2021-04-04T09:35:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:35:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:35:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:35:46.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:46.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:35:46.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench pool_to_remove/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:35:46.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:35:46.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:35:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2777.78 bytes/sec: 11 MiB/s 2021-04-04T09:35:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror test_remove_data_pool 100 2021-04-04T09:35:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:35:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:35:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:35:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:35:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:35:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:35:46.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/test_remove_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:35:46.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:35:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:35:46.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2777.78 bytes/sec: 11 MiB/s 2021-04-04T09:35:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 pool_to_remove test 2021-04-04T09:35:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:35:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=pool_to_remove 2021-04-04T09:35:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:35:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:35:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 pool_to_remove test 2021-04-04T09:35:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:35:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:35:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=pool_to_remove 2021-04-04T09:35:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:35:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:35:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:35:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:35:46.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:35:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 pool_to_remove test 2021-04-04T09:35:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=pool_to_remove 2021-04-04T09:35:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:35:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:35:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:35:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush pool_to_remove/test 2021-04-04T09:35:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:35:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:35:46.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:35:46.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:35:46.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:35:46.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:35:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:35:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:35:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:46.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:46.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:46.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:46.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:46.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:35:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:35:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:35:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:46.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:46.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:35:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:35:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:35:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:35:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:35:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:35:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:35:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:35:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:35:46.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:35:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:35:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:35:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:35:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:35:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:46.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:35:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:35:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:35:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:35:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:35:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:35:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:35:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:35:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:35:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:35:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:35:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:47.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:35:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:35:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:35:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:35:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:47.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:35:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:35:47.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:35:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:35:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:35:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:35:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:35:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:35:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:47.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:35:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:35:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:35:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:35:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:35:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:35:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:35:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:48.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:48.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:35:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:35:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:35:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:48.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:48.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:48.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:48.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:48.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:35:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:48.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:35:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:35:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:35:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:35:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:35:48.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:35:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:35:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:35:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:35:48.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:35:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:48.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:35:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:35:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:35:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:35:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:35:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:35:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:35:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:35:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:35:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:35:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:35:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:35:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:48.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:35:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:35:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:48.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:35:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:35:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:35:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:35:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:35:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:35:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:35:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:35:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:35:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:35:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:35:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:48.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:35:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:35:48.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:35:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:35:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:35:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:35:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:35:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:35:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:35:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:35:50.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:35:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:50.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:35:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:35:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:35:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:50.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:50.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:35:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:35:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:50.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:35:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:35:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:35:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:50.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:35:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:35:50.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:35:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:35:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:50.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:35:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:35:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:50.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:35:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:35:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:35:50.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:35:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:35:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:35:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:35:50.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:35:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:50.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:35:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:35:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:35:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:51.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:35:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:51.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:35:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:35:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:35:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:35:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:35:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:35:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:35:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:35:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:35:51.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:35:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:51.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:35:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:51.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:51.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:35:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:35:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 4 2021-04-04T09:35:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:35:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:35:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:35:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:35:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:35:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:35:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:35:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:35:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:35:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:55.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:35:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:35:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:35:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:35:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:35:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:35:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:35:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:55.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:35:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:35:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:35:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:35:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:35:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:35:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:35:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:35:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:35:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:35:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:35:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:35:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:35:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:35:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:35:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:35:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:35:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:35:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:35:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:35:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:35:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:35:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:35:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:35:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:35:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:35:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:35:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror flush pool_to_remove/test 2021-04-04T09:35:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:35:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:35:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:55.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:35:55.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:35:55.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:35:55.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:35:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:35:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:35:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:35:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:35:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:35:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:35:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:35:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:35:55.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:35:55.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:35:55.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:35:55.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:55.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:35:55.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:35:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:35:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:35:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:35:55.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:35:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:35:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:35:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:35:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:35:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:35:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:36:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:36:03.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:36:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush pool_to_remove/test 2021-04-04T09:36:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:36:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:36:03.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:36:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:03.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:36:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:36:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:36:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:36:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:36:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:36:03.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:03.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:03.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:36:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:36:03.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:36:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:03.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:36:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:36:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:36:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:36:03.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:36:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:36:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush pool_to_remove/test 2021-04-04T09:36:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:36:03.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:03.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:36:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:36:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:03.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:03.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:36:03.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:36:03.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:36:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:36:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:36:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:36:03.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:03.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:36:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:36:03.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:36:03.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:03.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:03.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:36:03.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:36:03.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:36:03.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:36:03.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:36:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:03.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:36:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror flush pool_to_remove/test 2021-04-04T09:36:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:36:03.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:36:03.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:36:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:03.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:36:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:36:03.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:36:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:36:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:36:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:36:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:03.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:03.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:36:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:36:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:36:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:03.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:36:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:36:03.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:36:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:36:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:36:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:36:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror flush pool_to_remove/test 2021-04-04T09:36:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:36:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:36:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:36:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:36:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:36:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:04.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:36:04.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:36:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:36:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:36:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:36:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:36:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:36:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:36:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:36:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush pool_to_remove/test 2021-04-04T09:36:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:36:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 pool_to_remove test 2021-04-04T09:36:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=pool_to_remove 2021-04-04T09:36:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:36:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 pool_to_remove test 2021-04-04T09:36:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=pool_to_remove 2021-04-04T09:36:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:36:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:36:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-pool_to_remove-test.status 2021-04-04T09:36:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-pool_to_remove-test.status 2021-04-04T09:36:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:36:04.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-pool_to_remove-test.status 2021-04-04T09:36:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + rbd --cluster cluster2 journaltee status /tmp/tmp.zj6T3cKcnO/cluster2-pool_to_remove-test.status --image 2021-04-04T09:36:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: pool_to_remove/test 2021-04-04T09:36:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:36:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:36:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:36:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:36:04.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=555b1bb4-9931-4f70-980d-a8772f0416ee, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:36:04.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-pool_to_remove-test.status 2021-04-04T09:36:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=3, tag_tid=2, entry_tid=3 2021-04-04T09:36:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 pool_to_remove test 2021-04-04T09:36:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=pool_to_remove 2021-04-04T09:36:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:36:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 pool_to_remove test ..* 2021-04-04T09:36:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=pool_to_remove 2021-04-04T09:36:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:36:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:36:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-pool_to_remove-test.status 2021-04-04T09:36:04.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-pool_to_remove-test.status 2021-04-04T09:36:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:36:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-pool_to_remove-test.status 2021-04-04T09:36:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image pool_to_remove/test 2021-04-04T09:36:04.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-pool_to_remove-test.status 2021-04-04T09:36:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:36:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:36:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:36:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:36:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=555b1bb4-9931-4f70-980d-a8772f0416ee, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:36:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-pool_to_remove-test.status 2021-04-04T09:36:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=3, tag_tid=2, entry_tid=3 2021-04-04T09:36:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=3, tag_tid=2, entry_tid=3 -a object_number=3, tag_tid=2, entry_tid=3 = object_number=3, tag_tid=2, entry_tid=3 2021-04-04T09:36:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:36:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 pool_to_remove test up+replaying primary_position 2021-04-04T09:36:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=pool_to_remove 2021-04-04T09:36:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:36:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:36:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:36:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:36:04.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:36:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 pool_to_remove test up+replaying primary_position 2021-04-04T09:36:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=pool_to_remove 2021-04-04T09:36:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:36:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:36:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:36:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:36:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-pool_to_remove-test.mirror_status 2021-04-04T09:36:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-pool_to_remove-test.mirror_status 2021-04-04T09:36:05.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:36:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-pool_to_remove-test.mirror_status 2021-04-04T09:36:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status pool_to_remove/test 2021-04-04T09:36:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-pool_to_remove-test.mirror_status 2021-04-04T09:36:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:36:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: d455a48f-f646-4a3b-877c-f5616b1628ec 2021-04-04T09:36:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:36:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:36:05.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.3 on smithi122 2021-04-04T09:36:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:35:56 2021-04-04T09:36:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:36:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:36:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:36:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:36:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:36:03 2021-04-04T09:36:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-pool_to_remove-test.mirror_status 2021-04-04T09:36:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:36:05.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .*primary_position /tmp/tmp.zj6T3cKcnO/cluster1-pool_to_remove-test.mirror_status 2021-04-04T09:36:05.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:36:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:36:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:36:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:36:05.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:36:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-pool_to_remove-test.mirror_status 2021-04-04T09:36:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2021-04-04T09:36:05.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-pool_to_remove-test.mirror_status 2021-04-04T09:36:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:35:56 2021-04-04T09:36:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 pool_to_remove test up+replaying 2021-04-04 09:35:56 2021-04-04T09:36:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=pool_to_remove 2021-04-04T09:36:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:36:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:36:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:35:56 2021-04-04T09:36:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-pool_to_remove.mirror_status 2021-04-04T09:36:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-pool_to_remove.mirror_status 2021-04-04T09:36:05.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:36:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-pool_to_remove.mirror_status 2021-04-04T09:36:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status pool_to_remove --verbose --format xml 2021-04-04T09:36:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:36:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:36:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:36:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:35:56 2021-04-04T09:36:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:36:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:36:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:36:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:36:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:36:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:36:05.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:36:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test_remove_data_pool 2021-04-04T09:36:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:36:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:05.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:36:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:36:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:36:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror test_remove_data_pool 2021-04-04T09:36:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:36:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:36:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:36:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:36:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:36:05.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:36:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:36:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror test_remove_data_pool 2021-04-04T09:36:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:36:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:36:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_remove_data_pool ] 2021-04-04T09:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/test_remove_data_pool 2021-04-04T09:36:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/test_remove_data_pool 2021-04-04T09:36:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:36:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:36:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:36:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:36:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:36:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:36:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/test_remove_data_pool 2021-04-04T09:36:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:36:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:36:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:36:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:36:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:36:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:36:05.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:36:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:05.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:36:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:36:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:36:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:36:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:36:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:36:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:36:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:36:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:36:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/test_remove_data_pool 2021-04-04T09:36:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_remove_data_pool'] 2021-04-04T09:36:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:36:05.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/test_remove_data_pool 2021-04-04T09:36:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:36:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:36:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:36:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:05.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:36:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:36:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:36:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:36:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:36:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:36:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:36:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:36:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:36:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:05.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:36:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:36:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:36:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:36:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/test_remove_data_pool 2021-04-04T09:36:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_remove_data_pool'] 2021-04-04T09:36:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:36:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror flush mirror/test_remove_data_pool 2021-04-04T09:36:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:36:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:36:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:36:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:36:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:36:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:36:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:36:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:36:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:36:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:36:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:36:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:36:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:36:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:36:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:36:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:36:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror flush mirror/test_remove_data_pool 2021-04-04T09:36:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test_remove_data_pool'] 2021-04-04T09:36:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:36:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror flush mirror/test_remove_data_pool 2021-04-04T09:36:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:36:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:36:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:36:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:06.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:36:06.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:36:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:36:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:36:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:06.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:36:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:36:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:36:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:36:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:36:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:36:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:06.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:36:06.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:36:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:36:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:36:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror flush mirror/test_remove_data_pool 2021-04-04T09:36:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:36:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror test_remove_data_pool 2021-04-04T09:36:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:36:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:36:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test_remove_data_pool 2021-04-04T09:36:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:36:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:36:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:36:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_remove_data_pool.status 2021-04-04T09:36:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_remove_data_pool.status 2021-04-04T09:36:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:36:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_remove_data_pool.status 2021-04-04T09:36:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test_remove_data_pool 2021-04-04T09:36:06.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_remove_data_pool.status 2021-04-04T09:36:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:36:06.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:36:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:36:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:36:06.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:36:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_remove_data_pool.status 2021-04-04T09:36:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=3, tag_tid=2, entry_tid=3 2021-04-04T09:36:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror test_remove_data_pool 2021-04-04T09:36:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:36:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:36:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror test_remove_data_pool ..* 2021-04-04T09:36:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:36:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:36:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:36:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror-test_remove_data_pool.status 2021-04-04T09:36:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror-test_remove_data_pool.status 2021-04-04T09:36:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:36:06.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_remove_data_pool.status 2021-04-04T09:36:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/test_remove_data_pool 2021-04-04T09:36:06.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_remove_data_pool.status 2021-04-04T09:36:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:36:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:36:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:36:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:36:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:36:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror-test_remove_data_pool.status 2021-04-04T09:36:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=3, tag_tid=2, entry_tid=3 2021-04-04T09:36:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=3, tag_tid=2, entry_tid=3 -a object_number=3, tag_tid=2, entry_tid=3 = object_number=3, tag_tid=2, entry_tid=3 2021-04-04T09:36:06.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:36:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test_remove_data_pool up+replaying primary_position 2021-04-04T09:36:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:36:06.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:36:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:36:06.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:36:06.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:36:06.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:36:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror test_remove_data_pool up+replaying primary_position 2021-04-04T09:36:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:36:07.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:36:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:36:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:36:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:36:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror-test_remove_data_pool.mirror_status 2021-04-04T09:36:07.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror-test_remove_data_pool.mirror_status 2021-04-04T09:36:07.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:36:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_remove_data_pool.mirror_status 2021-04-04T09:36:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status+ mirror/test_remove_data_pool 2021-04-04T09:36:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_remove_data_pool.mirror_status 2021-04-04T09:36:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test_remove_data_pool: 2021-04-04T09:36:07.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 19dd5345-25d8-4554-8034-6c31e0df778e 2021-04-04T09:36:07.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:36:07.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":35.0,"entries_behind_primary":0,"entries_per_second":0.4,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:36:07.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.3 on smithi122 2021-04-04T09:36:07.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:36:05 2021-04-04T09:36:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:36:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:36:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+stopped 2021-04-04T09:36:07.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: local image is primary 2021-04-04T09:36:07.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:36:02 2021-04-04T09:36:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_remove_data_pool.mirror_status 2021-04-04T09:36:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:36:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .*primary_position /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_remove_data_pool.mirror_status 2021-04-04T09:36:07.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":35.0,"entries_behind_primary":0,"entries_per_second":0.4,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:36:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:36:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:36:07.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:36:07.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:36:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_remove_data_pool.mirror_status 2021-04-04T09:36:07.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.3 on smithi122 2021-04-04T09:36:07.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror-test_remove_data_pool.mirror_status 2021-04-04T09:36:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:36:05 2021-04-04T09:36:07.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test_remove_data_pool up+replaying 2021-04-04 09:36:05 2021-04-04T09:36:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:36:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:36:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:36:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:36:05 2021-04-04T09:36:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror.mirror_status 2021-04-04T09:36:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror.mirror_status 2021-04-04T09:36:07.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:36:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror.mirror_status 2021-04-04T09:36:07.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2021-04-04T09:36:07.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:36:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:36:07.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_remove_data_pool']/last_update 2021-04-04T09:36:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:36:05 2021-04-04T09:36:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test_remove_data_pool']/state 2021-04-04T09:36:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:36:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:36:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:36:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:36:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:36:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:36:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool rm pool_to_remove pool_to_remove --yes-i-really-really-mean-it 2021-04-04T09:36:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:pool 'pool_to_remove' does not exist 2021-04-04T09:36:08.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool rm pool_to_remove pool_to_remove --yes-i-really-really-mean-it 2021-04-04T09:36:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:pool 'pool_to_remove' does not exist 2021-04-04T09:36:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_image_retry cluster2 mirror test_remove_data_pool 2021-04-04T09:36:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:36:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:36:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:36:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_image cluster2 mirror test_remove_data_pool 2021-04-04T09:36:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:36:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:36:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 snap purge mirror/test_remove_data_pool 2021-04-04T09:36:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 rm mirror/test_remove_data_pool 2021-04-04T09:36:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing image: 100% complete...done. 2021-04-04T09:36:10.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:36:10.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror test_remove_data_pool deleted 2021-04-04T09:36:10.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:10.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:36:10.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:36:10.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=deleted 2021-04-04T09:36:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:36:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:36:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:36:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test_remove_data_pool 2021-04-04T09:36:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:36:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:36:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test_remove_data_pool 2021-04-04T09:36:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:36:10.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test_remove_data_pool: (2) No such file or directory 2021-04-04T09:36:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id= 2021-04-04T09:36:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:36:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test_remove_data_pool deleted 2021-04-04T09:36:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:36:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:36:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=deleted 2021-04-04T09:36:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:36:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:36:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:36:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test_remove_data_pool 2021-04-04T09:36:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:36:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:36:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_remove_data_pool 2021-04-04T09:36:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test_remove_data_pool 2021-04-04T09:36:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:36:10.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test_remove_data_pool: (2) No such file or directory 2021-04-04T09:36:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:36:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:36:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test deleted = deleted 2021-04-04T09:36:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:36:10.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:36:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test 2021-04-04T09:36:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster2 2021-04-04T09:36:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:10.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster2 2021-04-04T09:36:10.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:36:10.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:36:10.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:36:10.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:36:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:36:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:36:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:10.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:36:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:36:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:36:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:36:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:10.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:10.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:10.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:36:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:10.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2021-04-04T09:36:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "pool_to_remove/test", 2021-04-04T09:36:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Stopped" 2021-04-04T09:36:10.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2021-04-04T09:36:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:36:10.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:36:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test 2021-04-04T09:36:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster2 2021-04-04T09:36:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster2 2021-04-04T09:36:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:36:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:36:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:36:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:36:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:36:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:36:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:36:12.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:12.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:12.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:12.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:12.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:36:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:36:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:36:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:12.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:12.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2021-04-04T09:36:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "pool_to_remove/test", 2021-04-04T09:36:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Stopped" 2021-04-04T09:36:12.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2021-04-04T09:36:12.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:36:12.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:36:14.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test 2021-04-04T09:36:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster2 2021-04-04T09:36:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster2 2021-04-04T09:36:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:36:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:36:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:36:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:36:14.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:36:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:36:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:36:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:14.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:36:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:36:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:36:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:14.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:36:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:14.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:14.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:14.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:14.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:36:14.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:36:14.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:36:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:36:14.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:14.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:36:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:14.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:36:14.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:36:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:36:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:36:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:36:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:14.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:14.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:36:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:14.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:36:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:36:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:36:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:36:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:36:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:36:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:36:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:36:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:36:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:36:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:36:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:14.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:14.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:36:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:36:14.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:36:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:36:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:36:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:36:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:36:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:36:14.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:36:14.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:14.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:14.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:14.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:36:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:14.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:14.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:36:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:14.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:36:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:36:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:36:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:36:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:36:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:36:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:15.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:15.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:36:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:36:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:36:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:15.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:36:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:15.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:15.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:36:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:36:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:36:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:36:15.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:36:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:36:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:36:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:36:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:36:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:36:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:36:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:15.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:36:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:36:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:36:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:36:15.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:36:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:36:15.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:36:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:36:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:36:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:36:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:36:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:15.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:36:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:36:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:36:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:36:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:36:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:36:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:36:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:36:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:36:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:16.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:36:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:36:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:16.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:16.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:16.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:36:18.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:36:18.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:36:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:36:18.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:36:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:36:18.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:36:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:36:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:18.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:36:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:18.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:18.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:18.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:36:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:36:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:36:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:36:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:36:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:36:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:36:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:36:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:36:18.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:36:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:36:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:36:18.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:18.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:18.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:18.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:18.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:36:18.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:36:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:36:18.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:36:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:36:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:36:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:36:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:18.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:36:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:36:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:18.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:36:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:36:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:18.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:36:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:36:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:18.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:36:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:36:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:36:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:36:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:36:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:18.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:36:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:36:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:36:18.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:18.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:36:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:18.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:18.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 4 2021-04-04T09:36:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:36:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:36:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:36:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:36:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:36:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:36:22.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:36:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:36:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:22.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:22.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:22.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:22.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:22.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:22.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:22.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:22.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:22.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:36:22.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:22.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:22.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:36:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:36:22.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:36:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:36:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:22.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:36:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:36:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:36:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:36:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:36:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:36:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:36:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:36:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:36:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:36:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:36:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:36:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:36:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:36:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:36:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:36:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:36:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:22.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:36:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:23.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:36:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:36:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:36:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:36:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:36:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:36:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:36:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:36:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:36:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:36:23.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:23.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:36:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:23.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:36:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:36:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:36:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:36:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:36:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:36:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:31.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:31.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:36:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:36:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:36:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:36:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:31.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:31.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:36:31.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:36:31.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:36:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:36:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:36:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:36:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:36:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:36:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:36:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:31.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:36:31.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:31.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:31.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:31.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:36:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:31.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:36:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:36:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:36:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:36:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:31.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:36:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:31.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:31.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:31.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:36:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:36:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:31.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:31.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:36:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:36:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:36:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:36:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:36:31.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:36:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:36:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:36:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:36:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:36:31.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:31.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:31.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:31.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:31.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:36:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:31.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:31.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:31.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:31.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:36:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:36:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:36:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:36:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:36:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:36:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:36:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:36:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:39.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:36:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:36:39.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:39.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:36:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:36:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:36:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:36:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:36:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:36:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:36:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:36:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:36:39.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:36:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:36:39.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:39.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:36:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:36:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:36:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:36:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:36:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:36:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:36:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:36:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:36:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:36:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:36:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:40.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:36:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:36:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:36:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:36:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:36:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:40.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:36:40.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:36:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:36:40.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:36:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:40.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:36:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:36:40.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:40.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:36:48.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:36:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:36:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:36:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:36:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:36:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:36:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:36:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:36:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:48.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:48.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:48.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:36:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:36:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:36:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:36:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:36:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:36:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:36:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:36:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:36:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:36:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:36:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:36:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:48.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:36:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:36:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:36:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:36:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:36:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:36:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:36:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:36:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:36:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:36:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:48.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:36:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:36:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:36:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:36:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:36:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:36:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:36:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:36:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:36:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:48.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:36:56.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:36:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:36:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:36:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:36:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:36:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:56.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:56.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:56.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:56.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:36:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:36:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:36:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:36:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:56.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:56.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:36:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:36:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:36:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:36:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:36:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:56.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:36:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:36:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:56.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:56.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:36:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:36:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:56.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:36:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:36:56.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:36:56.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:56.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:36:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:36:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:36:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:36:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:36:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:36:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:36:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:56.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:36:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:36:56.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:36:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:36:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:36:56.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:56.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:56.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:56.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:56.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:36:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:36:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:36:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:36:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:56.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:36:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:36:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:36:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:36:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:36:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:36:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:36:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:36:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:36:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:36:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:36:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:36:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:36:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:36:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:36:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:36:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:36:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:36:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:36:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:36:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:36:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:36:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:36:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:36:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:36:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:36:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:36:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:36:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:36:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:36:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:36:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:36:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:36:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:37:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:37:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:37:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:37:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:37:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:05.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:37:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:37:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:37:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:37:05.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:37:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:37:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:05.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:37:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:37:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:37:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:05.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:05.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:37:05.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:05.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:37:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:37:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:37:05.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:37:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:05.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:37:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:37:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:37:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:37:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:37:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:37:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:37:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:37:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:37:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:37:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:37:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:37:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:37:05.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:37:05.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:37:05.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:37:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:37:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:37:05.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:37:05.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:37:05.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:05.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:05.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:37:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:37:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:37:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:37:05.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:37:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:37:05.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:05.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:37:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:37:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:37:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:37:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:37:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:37:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:37:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:37:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:05.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:37:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:37:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:05.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:05.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:37:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:37:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:37:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:37:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:05.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:37:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:37:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:37:05.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:37:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:37:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:37:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:37:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:37:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:37:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:37:13.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:37:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:37:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:37:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:37:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:37:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:37:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:37:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:37:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:37:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:37:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:37:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:37:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:37:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:37:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:37:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:37:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:37:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:37:13.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:37:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:37:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:37:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:13.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:37:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:37:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:37:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:37:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:37:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:37:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:13.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:13.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:13.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:13.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:37:13.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:13.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:13.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:37:13.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:13.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:37:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:37:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:37:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:37:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:13.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:37:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:37:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:37:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:37:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:37:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:37:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:37:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:37:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:37:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:37:13.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:37:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:37:13.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:37:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:37:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:37:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:37:13.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:14.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:37:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:37:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:14.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:37:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:37:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:14.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:37:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:37:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:37:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:37:14.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:37:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:37:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:37:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:37:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:14.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:37:14.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:37:14.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:37:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:37:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:14.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:14.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:37:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:37:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:37:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:37:22.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:37:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:37:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:37:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:37:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:37:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:37:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:37:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:37:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:37:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:37:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:37:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:37:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:37:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:37:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:37:22.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:22.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:37:22.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:37:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:22.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:37:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:37:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:22.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:22.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:37:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:37:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:37:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:37:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:37:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:37:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:22.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:22.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:37:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:22.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:37:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:37:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:37:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:37:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:37:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:22.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:37:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:37:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:37:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:37:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:37:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:37:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:37:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:37:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:37:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:37:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:37:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:37:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:37:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:37:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:37:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:37:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:37:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:37:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:37:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:37:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:37:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:22.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:37:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:37:22.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:37:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:22.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:37:22.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:37:22.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:22.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:37:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:37:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:22.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:37:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:37:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:37:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:37:22.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:22.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:22.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:37:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:37:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:37:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:37:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:37:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:37:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:37:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:37:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:37:30.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:37:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:37:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:37:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:37:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:37:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:37:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:37:30.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:37:30.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:37:30.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:37:30.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:37:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:37:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:37:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:37:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:37:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:37:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:37:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:37:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:37:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:37:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:37:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:37:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:37:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:37:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:37:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:37:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:30.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:37:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:37:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:30.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:37:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:37:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:37:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:37:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:37:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:37:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:37:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:37:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:37:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:37:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:37:30.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:37:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:37:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:37:30.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:31.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:37:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:37:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:37:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:37:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:37:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:31.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:37:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:37:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:37:31.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:37:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:37:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:31.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:37:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:37:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:31.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:37:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:37:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:37:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:37:31.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:31.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 16 2021-04-04T09:37:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:37:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:37:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:37:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:37:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:37:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:37:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:47.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:37:47.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:37:47.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:37:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:37:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:47.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:37:47.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:37:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:47.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:37:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:37:47.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:37:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:37:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:37:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:37:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:37:47.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:37:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:37:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:37:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:37:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:37:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:37:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:37:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:37:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:37:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:37:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:37:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:37:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:37:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:37:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:37:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:37:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:37:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:47.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:37:47.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:37:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:37:47.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:37:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:37:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:37:47.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:37:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:37:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:37:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:37:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:37:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:37:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:37:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:37:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:37:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:37:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:37:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:37:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:37:47.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:37:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:37:47.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:37:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:37:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:47.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:47.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:37:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:37:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:37:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:37:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:37:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:37:47.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:37:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:37:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:37:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:37:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:37:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:37:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:37:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:37:47.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:37:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:37:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:37:47.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:37:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:37:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:37:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:37:47.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:37:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:37:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:37:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:37:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 16 2021-04-04T09:38:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:38:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:38:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:38:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:38:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:38:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:38:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:38:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:38:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:03.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:03.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:38:03.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:38:03.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:38:03.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:03.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:03.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:38:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:03.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:03.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:38:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:38:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:38:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:38:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:03.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:03.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:38:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:38:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:38:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:38:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:38:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:03.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:03.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:38:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:03.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:03.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:38:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:38:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:38:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:38:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:04.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:38:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:38:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:38:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:04.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:38:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:04.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 1 2021-04-04T09:38:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:38:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster2 rbd mirror status pool_to_remove/test 2021-04-04T09:38:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster2 2021-04-04T09:38:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster2 2021-04-04T09:38:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:38:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:38:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:38:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:38:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:38:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:38:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:38:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:38:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:38:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:38:04.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:04.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:04.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:38:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:04.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:04.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:38:04.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:38:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:04.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:38:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:38:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:38:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:38:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:04.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:04.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:38:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:04.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:04.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:38:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:38:04.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:38:04.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:04.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:04.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:38:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:38:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:04.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:04.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:38:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:04.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:04.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:38:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:04.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:38:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:38:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:38:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:04.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:04.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:38:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:38:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:38:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:04.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:04.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:38:04.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:04.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:04.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:38:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:38:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:38:05.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:38:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:38:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:38:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:38:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:38:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:05.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:38:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:38:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:05.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:05.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:38:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:38:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:38:05.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:05.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:38:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:38:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:38:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:05.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:38:05.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:05.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:05.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:05.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:38:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:38:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:38:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:05.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:38:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:38:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:38:05.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:38:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:38:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:38:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:38:06.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:06.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:06.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:06.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:38:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:38:06.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:06.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:06.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:38:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:06.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:38:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:38:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:38:08.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:38:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:38:08.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:38:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:38:08.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:08.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:08.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:38:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:38:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:38:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:38:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:38:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:38:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:38:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:38:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:38:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:38:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:38:08.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:08.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:08.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:08.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:38:08.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:38:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:38:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:38:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:38:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:38:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:08.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:08.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:38:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:38:08.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:38:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:38:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:38:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:38:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:38:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:38:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:08.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 4 2021-04-04T09:38:12.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:38:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:38:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:38:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:38:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:38:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:12.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:12.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:12.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:38:12.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:38:12.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:38:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:12.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:12.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:38:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:12.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:12.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:38:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:38:12.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:38:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:12.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:38:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:38:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:38:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:12.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:38:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:38:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:38:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:38:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:38:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:38:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:38:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:38:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:13.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:13.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:38:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:38:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:38:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:38:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:38:13.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:13.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:13.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:38:13.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:13.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:13.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:13.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:38:13.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:13.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:38:21.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:38:21.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:38:21.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:38:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:38:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:38:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:38:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:38:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:38:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:38:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:21.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:21.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:38:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:38:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:38:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:21.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:38:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:38:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:21.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:38:21.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:21.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:38:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:21.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:38:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:38:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:38:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:38:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:38:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:38:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:38:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:38:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:38:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:38:21.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:38:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:38:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:38:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:21.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:21.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:38:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:21.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:21.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:38:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:38:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:38:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:38:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:38:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:38:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:38:29.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:38:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:38:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:38:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:29.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:38:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:38:29.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:38:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:29.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:29.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:38:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:38:29.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:38:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:29.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:29.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:29.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:38:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:38:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:38:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:29.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:38:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:38:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:38:29.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:38:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:29.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:38:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:30.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:30.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:38:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:38:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:38:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:38:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:38:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:38:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:38:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:38:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:38:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:38:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:38:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:38:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:38:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:38.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:38.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:38:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:38:38.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:38:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:38.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:38:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:38.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:38:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:38:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:38:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:38:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:38:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:38:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:38.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:38:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:38:38.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:38:38.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:38.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:38:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:38.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:38.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:38.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:38:38.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:38:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:38:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:38.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:38:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:38:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:38:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:38:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:38.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:38.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:38.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:38.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:38.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:38:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:38:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:38:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:38:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:38:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:38:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:38:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:38:46.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:38:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:38:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:46.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:38:46.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:38:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:38:46.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:46.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:46.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:38:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:46.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:38:46.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:38:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:38:46.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:46.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:46.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:38:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:38:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:46.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:38:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:46.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:46.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:38:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:46.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:46.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:46.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:46.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:38:46.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:38:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:38:46.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:46.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:46.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:38:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:38:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:38:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:46.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:38:46.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:46.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:46.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:38:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:38:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:38:47.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:47.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:47.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:47.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:38:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:38:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:38:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:47.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:47.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:38:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:47.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:38:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:38:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:38:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:38:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:38:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:38:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:55.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:38:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:38:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:38:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:38:55.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:55.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:55.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:55.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:38:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:38:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:38:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:38:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:38:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:55.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:38:55.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:38:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:55.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:38:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:38:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:38:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:38:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:38:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:38:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:38:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:38:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:38:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:55.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:38:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:38:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:38:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:38:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:38:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:38:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:38:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:38:55.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:55.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:55.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:38:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:38:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:38:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:38:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:55.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:38:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:55.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:38:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:38:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:38:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:38:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:38:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:38:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:38:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:38:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:38:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:38:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:38:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:38:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:38:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:38:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:38:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:38:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:38:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:38:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:38:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:38:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:38:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:55.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:55.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:38:55.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:38:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:38:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:38:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:39:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:39:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:39:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:39:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:39:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:39:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:39:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:39:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:39:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:39:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:39:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:39:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:39:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:39:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:03.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:39:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:03.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:39:03.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:39:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:39:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:39:03.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:39:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:39:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:39:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:39:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:39:03.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:03.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:39:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:39:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:39:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:39:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:39:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:39:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:39:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:39:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:39:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:39:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:39:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:03.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:39:03.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:39:03.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:03.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:39:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:39:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:39:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:39:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:39:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:39:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:39:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:39:03.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:03.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:03.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:03.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:03.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:39:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:39:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:03.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:39:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:39:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:39:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:39:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:39:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:39:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:39:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:39:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:39:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:39:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:39:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:39:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:39:04.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:39:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:04.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:39:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:39:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:39:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:39:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:39:04.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:39:04.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:39:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:39:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:39:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:39:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:39:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:39:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:39:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:12.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:39:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:39:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:39:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:39:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:12.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:39:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:39:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:12.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:39:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:12.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:39:12.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:12.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:39:12.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:39:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:39:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:39:12.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:12.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:12.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:39:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:39:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:39:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:39:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:39:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:39:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:39:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:39:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:39:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:39:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:39:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:12.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:39:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:12.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:39:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:39:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:39:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:39:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:12.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:39:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:39:12.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:39:12.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:39:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:12.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:39:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:39:12.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:39:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:39:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:39:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:39:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:39:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:39:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:39:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:39:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:39:12.432 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:39:12.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:12.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:39:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:39:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:39:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:39:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:12.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:12.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:12.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:12.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:39:12.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:12.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:12.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:39:12.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:39:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:39:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:12.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:12.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:39:12.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:39:12.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:39:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:12.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:12.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:39:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:39:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:39:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:39:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:39:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:39:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:39:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:39:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:39:20.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:39:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:20.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:39:20.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:39:20.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:20.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:39:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:20.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:20.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:39:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:20.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:39:20.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:39:20.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:20.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:39:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:39:20.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:39:20.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:39:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:20.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:39:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:39:20.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:39:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:39:20.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:39:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:39:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:20.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:39:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:39:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:39:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:39:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:39:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:39:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:39:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:39:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:20.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:39:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:39:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:39:20.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:39:20.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:39:20.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:39:20.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:39:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:39:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:39:20.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:39:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:39:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:39:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:39:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:39:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:39:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:39:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:20.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:39:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:39:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:39:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:39:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:39:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:39:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:39:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:39:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:21.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:39:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:39:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:39:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:39:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:21.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:39:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:39:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:21.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:21.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:39:21.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:39:21.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:21.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:39:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:39:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:39:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:39:21.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:39:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:39:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 16 2021-04-04T09:39:37.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:39:37.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:39:37.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:39:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:39:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:37.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:39:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:39:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:39:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:39:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:39:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:39:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:37.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:39:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:39:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:39:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:37.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:39:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:39:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:39:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:39:37.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:39:37.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:37.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:39:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:39:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:37.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:39:37.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:39:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:37.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:39:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:39:37.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:39:37.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:39:37.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:39:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:39:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:37.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:39:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:39:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:37.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:39:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:39:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:39:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:39:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:37.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:37.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:39:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:39:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:39:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:39:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:39:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:39:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:39:37.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:39:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:39:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:39:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:37.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:39:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:37.430 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:39:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:39:37.434 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:39:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:39:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:39:37.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:39:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:39:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:37.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:39:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:39:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:39:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:39:37.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:39:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:39:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:39:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:39:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:37.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:39:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:37.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:39:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:39:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:39:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:39:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:39:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 16 2021-04-04T09:39:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster2 != cluster2 ] 2021-04-04T09:39:53.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:39:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:0 rbd mirror status pool_to_remove/test 2021-04-04T09:39:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:39:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:53.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:53.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:39:53.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:39:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:53.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:53.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:53.670 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:53.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:39:53.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:53.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:53.671 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:39:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:39:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:39:53.672 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:39:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:39:53.673 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:53.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:39:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:39:53.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:39:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:39:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:39:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:39:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:39:53.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:1 rbd mirror status pool_to_remove/test 2021-04-04T09:39:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:39:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:39:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:39:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:39:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:39:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:39:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:39:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:39:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:39:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:39:53.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:39:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:39:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:39:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:39:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:39:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:39:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:2 rbd mirror status pool_to_remove/test 2021-04-04T09:39:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:39:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:39:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:39:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:39:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:39:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:39:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:39:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:39:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:39:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:53.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:39:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:39:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:39:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:39:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:39:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:39:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:39:53.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:54.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster2:3 rbd mirror status pool_to_remove/test 2021-04-04T09:39:54.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:39:54.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:39:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:39:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:39:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:39:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:54.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:39:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:39:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:39:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:39:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:39:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:39:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:39:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:39:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:39:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:39:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:39:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:39:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:39:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:39:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:39:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:39:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:39:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:39:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:39:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:39:54.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok rbd mirror status pool_to_remove/test 2021-04-04T09:39:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['pool_to_remove/test'] 2021-04-04T09:39:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:39:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:39:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 1 2021-04-04T09:39:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog TEST: snapshot rename 2021-04-04T09:39:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:39:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:39:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:39:54 TEST: snapshot rename 2021-04-04T09:39:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:39:54 TEST: snapshot rename 2021-04-04T09:39:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ snap_name=snap_rename 2021-04-04T09:39:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_snapshot cluster2 mirror test2 snap_rename_0 2021-04-04T09:39:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_0 2021-04-04T09:39:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap create mirror/test2@snap_rename_0 2021-04-04T09:39:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2021-04-04T09:39:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 1 20 2021-04-04T09:39:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 1 - 1 2021-04-04T09:39:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_0 snap_rename_1 2021-04-04T09:39:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_0 2021-04-04T09:39:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_1 2021-04-04T09:39:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_0 mirror/test2@snap_rename_1 2021-04-04T09:39:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 2 - 1 2021-04-04T09:39:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_1 snap_rename_2 2021-04-04T09:39:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_1 2021-04-04T09:39:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_2 2021-04-04T09:39:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_1 mirror/test2@snap_rename_2 2021-04-04T09:39:55.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 3 - 1 2021-04-04T09:39:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_2 snap_rename_3 2021-04-04T09:39:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:55.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_2 2021-04-04T09:39:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_3 2021-04-04T09:39:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_2 mirror/test2@snap_rename_3 2021-04-04T09:39:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 4 - 1 2021-04-04T09:39:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_3 snap_rename_4 2021-04-04T09:39:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_3 2021-04-04T09:39:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_4 2021-04-04T09:39:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_3 mirror/test2@snap_rename_4 2021-04-04T09:39:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 5 - 1 2021-04-04T09:39:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_4 snap_rename_5 2021-04-04T09:39:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_4 2021-04-04T09:39:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_5 2021-04-04T09:39:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_4 mirror/test2@snap_rename_5 2021-04-04T09:39:55.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 6 - 1 2021-04-04T09:39:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_5 snap_rename_6 2021-04-04T09:39:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:55.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_5 2021-04-04T09:39:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_6 2021-04-04T09:39:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_5 mirror/test2@snap_rename_6 2021-04-04T09:39:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 7 - 1 2021-04-04T09:39:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_6 snap_rename_7 2021-04-04T09:39:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_6 2021-04-04T09:39:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_7 2021-04-04T09:39:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_6 mirror/test2@snap_rename_7 2021-04-04T09:39:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 8 - 1 2021-04-04T09:39:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_7 snap_rename_8 2021-04-04T09:39:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:55.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_7 2021-04-04T09:39:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_8 2021-04-04T09:39:55.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_7 mirror/test2@snap_rename_8 2021-04-04T09:39:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 9 - 1 2021-04-04T09:39:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_8 snap_rename_9 2021-04-04T09:39:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_8 2021-04-04T09:39:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_9 2021-04-04T09:39:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_8 mirror/test2@snap_rename_9 2021-04-04T09:39:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 10 - 1 2021-04-04T09:39:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_9 snap_rename_10 2021-04-04T09:39:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:55.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_9 2021-04-04T09:39:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_10 2021-04-04T09:39:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_9 mirror/test2@snap_rename_10 2021-04-04T09:39:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 11 - 1 2021-04-04T09:39:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_10 snap_rename_11 2021-04-04T09:39:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_10 2021-04-04T09:39:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_11 2021-04-04T09:39:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_10 mirror/test2@snap_rename_11 2021-04-04T09:39:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 12 - 1 2021-04-04T09:39:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_11 snap_rename_12 2021-04-04T09:39:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_11 2021-04-04T09:39:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_12 2021-04-04T09:39:55.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_11 mirror/test2@snap_rename_12 2021-04-04T09:39:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 13 - 1 2021-04-04T09:39:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_12 snap_rename_13 2021-04-04T09:39:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_12 2021-04-04T09:39:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_13 2021-04-04T09:39:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_12 mirror/test2@snap_rename_13 2021-04-04T09:39:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 14 - 1 2021-04-04T09:39:56.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_13 snap_rename_14 2021-04-04T09:39:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_13 2021-04-04T09:39:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_14 2021-04-04T09:39:56.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_13 mirror/test2@snap_rename_14 2021-04-04T09:39:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 15 - 1 2021-04-04T09:39:56.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_14 snap_rename_15 2021-04-04T09:39:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_14 2021-04-04T09:39:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_15 2021-04-04T09:39:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_14 mirror/test2@snap_rename_15 2021-04-04T09:39:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 16 - 1 2021-04-04T09:39:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_15 snap_rename_16 2021-04-04T09:39:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_15 2021-04-04T09:39:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_16 2021-04-04T09:39:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_15 mirror/test2@snap_rename_16 2021-04-04T09:39:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 17 - 1 2021-04-04T09:39:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_16 snap_rename_17 2021-04-04T09:39:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_16 2021-04-04T09:39:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_17 2021-04-04T09:39:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_16 mirror/test2@snap_rename_17 2021-04-04T09:39:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 18 - 1 2021-04-04T09:39:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_17 snap_rename_18 2021-04-04T09:39:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:56.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_17 2021-04-04T09:39:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_18 2021-04-04T09:39:56.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_17 mirror/test2@snap_rename_18 2021-04-04T09:39:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 19 - 1 2021-04-04T09:39:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_18 snap_rename_19 2021-04-04T09:39:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:56.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_18 2021-04-04T09:39:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_19 2021-04-04T09:39:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_18 mirror/test2@snap_rename_19 2021-04-04T09:39:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ expr 20 - 1 2021-04-04T09:39:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rename_snapshot cluster2 mirror test2 snap_rename_19 snap_rename_20 2021-04-04T09:39:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:39:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap=snap_rename_19 2021-04-04T09:39:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local new_snap=snap_rename_20 2021-04-04T09:39:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 snap rename mirror/test2@snap_rename_19 mirror/test2@snap_rename_20 2021-04-04T09:39:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_snap_present cluster1 mirror test2 snap_rename_20 2021-04-04T09:39:56.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:39:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:39:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test2 2021-04-04T09:39:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local snap_name=snap_rename_20 2021-04-04T09:39:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:39:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:39:57.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2021-04-04T09:39:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error setting snapshot context: (2) No such file or directory 2021-04-04T09:39:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ continue 2021-04-04T09:39:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:39:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2021-04-04T09:39:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error setting snapshot context: (2) No such file or directory 2021-04-04T09:39:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ continue 2021-04-04T09:39:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 4 2021-04-04T09:40:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2021-04-04T09:40:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error setting snapshot context: (2) No such file or directory 2021-04-04T09:40:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ continue 2021-04-04T09:40:03.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:40:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2021-04-04T09:40:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error setting snapshot context: (2) No such file or directory 2021-04-04T09:40:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ continue 2021-04-04T09:40:11.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:40:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 info mirror/test2@snap_rename_20 2021-04-04T09:40:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test2': 2021-04-04T09:40:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:40:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:40:19.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 3 2021-04-04T09:40:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0b93cfe9e 2021-04-04T09:40:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0b93cfe9e 2021-04-04T09:40:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:40:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:40:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:40:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:40:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:35:24 2021 2021-04-04T09:40:19.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:35:24 2021 2021-04-04T09:40:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:35:24 2021 2021-04-04T09:40:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: protected: False 2021-04-04T09:40:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10e0b93cfe9e 2021-04-04T09:40:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:40:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:40:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 80ed7469-0640-4a03-9952-a34e4d433832 2021-04-04T09:40:19.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:40:19.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog TEST: disable mirror while daemon is stopped 2021-04-04T09:40:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:40:19.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:40:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:40:19 TEST: disable mirror while daemon is stopped 2021-04-04T09:40:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:40:19 TEST: disable mirror while daemon is stopped 2021-04-04T09:40:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirrors cluster1 2021-04-04T09:40:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2021-04-04T09:40:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:40:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster1:0 2021-04-04T09:40:19.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:40:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2021-04-04T09:40:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid 2021-04-04T09:40:19.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:0 2021-04-04T09:40:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:40:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:40:19.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:19.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 pid file 2021-04-04T09:40:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2021-04-04T09:40:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2021-04-04T09:40:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pid=15736 2021-04-04T09:40:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n 15736 ] 2021-04-04T09:40:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ kill 15736 2021-04-04T09:40:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:40:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=15736 $2 == pid {print; exit 1} 2021-04-04T09:40:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:40:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:40:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=15736 $2 == pid {print; exit 1} 2021-04-04T09:40:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:40:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:40:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:40:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:20.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:40:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster2 2021-04-04T09:40:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:40:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:40:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:40:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:0 2021-04-04T09:40:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:40:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:40:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:21.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:21.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 pid file 2021-04-04T09:40:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2021-04-04T09:40:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2021-04-04T09:40:21.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster1:1 2021-04-04T09:40:21.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:40:21.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2021-04-04T09:40:21.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid 2021-04-04T09:40:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:1 2021-04-04T09:40:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:40:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:21.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:40:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:40:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:40:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 pid file 2021-04-04T09:40:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2021-04-04T09:40:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2021-04-04T09:40:21.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pid=15764 2021-04-04T09:40:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n 15764 ] 2021-04-04T09:40:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ kill 15764 2021-04-04T09:40:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:40:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=15764 $2 == pid {print; exit 1} 2021-04-04T09:40:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:40:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:40:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=15764 $2 == pid {print; exit 1} 2021-04-04T09:40:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:40:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:40:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:40:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:40:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:22.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:40:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:40:22.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:40:22.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:40:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster2 2021-04-04T09:40:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:40:22.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:40:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:40:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:22.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:40:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:22.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:22.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:40:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:40:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:40:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:1 2021-04-04T09:40:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:40:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:40:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:40:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:22.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:40:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 pid file 2021-04-04T09:40:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2021-04-04T09:40:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2021-04-04T09:40:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster1:2 2021-04-04T09:40:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:40:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2021-04-04T09:40:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid 2021-04-04T09:40:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:2 2021-04-04T09:40:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:40:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:40:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:40:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:22.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:22.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 pid file 2021-04-04T09:40:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2021-04-04T09:40:22.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2021-04-04T09:40:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pid=15792 2021-04-04T09:40:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n 15792 ] 2021-04-04T09:40:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ kill 15792 2021-04-04T09:40:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:23.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:40:23.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=15792 $2 == pid {print; exit 1} 2021-04-04T09:40:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:40:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:40:23.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=15792 $2 == pid {print; exit 1} 2021-04-04T09:40:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:40:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:40:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:40:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:40:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:23.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:40:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:23.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:40:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:40:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:40:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster2 2021-04-04T09:40:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:40:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:40:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:40:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:40:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:23.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:40:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:40:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:40:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:2 2021-04-04T09:40:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:40:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:40:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:40:23.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:40:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 pid file 2021-04-04T09:40:23.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2021-04-04T09:40:23.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2021-04-04T09:40:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster1:3 2021-04-04T09:40:23.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:40:23.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2021-04-04T09:40:23.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:23.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid 2021-04-04T09:40:23.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:3 2021-04-04T09:40:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:40:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:40:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:40:23.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:40:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:40:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 pid file 2021-04-04T09:40:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2021-04-04T09:40:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2021-04-04T09:40:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pid=15852 2021-04-04T09:40:23.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n 15852 ] 2021-04-04T09:40:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ kill 15852 2021-04-04T09:40:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:40:24.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=15852 $2 == pid {print; exit 1} 2021-04-04T09:40:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:40:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:40:24.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=15852 $2 == pid {print; exit 1} 2021-04-04T09:40:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:40:24.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:40:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:40:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:40:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:24.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:40:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:40:24.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:24.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:40:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:40:24.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:40:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster2 2021-04-04T09:40:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:40:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:40:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:40:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:40:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:40:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:24.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:40:24.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:40:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:40:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:3 2021-04-04T09:40:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:40:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:40:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:40:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:40:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:40:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 pid file 2021-04-04T09:40:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2021-04-04T09:40:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2021-04-04T09:40:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirrors cluster2 2021-04-04T09:40:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2021-04-04T09:40:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:40:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster2:0 2021-04-04T09:40:24.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:40:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2021-04-04T09:40:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid 2021-04-04T09:40:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster2:0 2021-04-04T09:40:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:40:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:40:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:40:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:40:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:40:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 pid file 2021-04-04T09:40:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2021-04-04T09:40:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2021-04-04T09:40:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pid=19137 2021-04-04T09:40:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n 19137 ] 2021-04-04T09:40:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ kill 19137 2021-04-04T09:40:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:40:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=19137 $2 == pid {print; exit 1} 2021-04-04T09:40:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:40:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:40:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=19137 $2 == pid {print; exit 1} 2021-04-04T09:40:25.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster1 2021-04-04T09:40:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:40:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:25.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:40:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:40:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:25.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:40:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:40:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:40:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:40:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:40:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:40:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:0 cluster2 2021-04-04T09:40:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:0 2021-04-04T09:40:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 local_cluster instance 2021-04-04T09:40:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:40:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:40:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:40:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:40:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 admin socket 2021-04-04T09:40:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:40:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2021-04-04T09:40:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster2:0 2021-04-04T09:40:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:0 2021-04-04T09:40:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:0 cluster instance 2021-04-04T09:40:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:0 2021-04-04T09:40:25.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster2:0 ] 2021-04-04T09:40:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:40:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:25.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.0 pid file 2021-04-04T09:40:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2021-04-04T09:40:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2021-04-04T09:40:25.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster2:1 2021-04-04T09:40:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:40:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2021-04-04T09:40:25.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid 2021-04-04T09:40:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster2:1 2021-04-04T09:40:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:40:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:40:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:40:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:25.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:40:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:40:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 pid file 2021-04-04T09:40:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2021-04-04T09:40:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2021-04-04T09:40:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pid=19165 2021-04-04T09:40:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n 19165 ] 2021-04-04T09:40:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ kill 19165 2021-04-04T09:40:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:40:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=19165 $2 == pid {print; exit 1} 2021-04-04T09:40:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:40:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:40:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=19165 $2 == pid {print; exit 1} 2021-04-04T09:40:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster1 2021-04-04T09:40:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:40:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:26.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:40:26.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:40:26.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:26.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:26.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:40:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:40:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:40:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:40:26.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:40:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:40:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:1 cluster2 2021-04-04T09:40:26.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:1 2021-04-04T09:40:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 local_cluster instance 2021-04-04T09:40:26.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:40:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:40:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:40:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:40:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 admin socket 2021-04-04T09:40:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:40:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.asok 2021-04-04T09:40:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster2:1 2021-04-04T09:40:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:1 2021-04-04T09:40:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:1 cluster instance 2021-04-04T09:40:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:1 2021-04-04T09:40:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster2:1 ] 2021-04-04T09:40:26.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:40:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.1 pid file 2021-04-04T09:40:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2021-04-04T09:40:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2021-04-04T09:40:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster2:2 2021-04-04T09:40:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:40:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2021-04-04T09:40:26.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:26.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid 2021-04-04T09:40:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster2:2 2021-04-04T09:40:26.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:40:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:40:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:40:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:26.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:26.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:40:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:40:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:26.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 pid file 2021-04-04T09:40:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2021-04-04T09:40:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2021-04-04T09:40:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pid=19209 2021-04-04T09:40:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n 19209 ] 2021-04-04T09:40:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ kill 19209 2021-04-04T09:40:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:40:27.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=19209 $2 == pid {print; exit 1} 2021-04-04T09:40:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:40:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:40:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=19209 $2 == pid {print; exit 1} 2021-04-04T09:40:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster1 2021-04-04T09:40:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:40:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:40:27.676 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:40:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:40:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:40:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:40:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:40:27.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:40:27.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:40:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:2 cluster2 2021-04-04T09:40:27.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:2 2021-04-04T09:40:27.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:27.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:27.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 local_cluster instance 2021-04-04T09:40:27.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:40:27.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:27.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:27.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:40:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:40:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:40:27.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 admin socket 2021-04-04T09:40:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:40:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.asok 2021-04-04T09:40:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster2:2 2021-04-04T09:40:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:2 2021-04-04T09:40:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:2 cluster instance 2021-04-04T09:40:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:2 2021-04-04T09:40:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster2:2 ] 2021-04-04T09:40:27.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:40:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:27.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.2 pid file 2021-04-04T09:40:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2021-04-04T09:40:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2021-04-04T09:40:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ stop_mirror cluster2:3 2021-04-04T09:40:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:40:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local sig= 2021-04-04T09:40:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid 2021-04-04T09:40:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster2:3 2021-04-04T09:40:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:40:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:40:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:40:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:40:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:40:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:40:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 pid file 2021-04-04T09:40:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2021-04-04T09:40:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2021-04-04T09:40:27.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pid=19291 2021-04-04T09:40:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n 19291 ] 2021-04-04T09:40:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ kill 19291 2021-04-04T09:40:27.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:28.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:40:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=19291 $2 == pid {print; exit 1} 2021-04-04T09:40:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ break 2021-04-04T09:40:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:40:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=19291 $2 == pid {print; exit 1} 2021-04-04T09:40:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster1 2021-04-04T09:40:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:40:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:40:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:40:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:40:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:40:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:40:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:40:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:40:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:40:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:40:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster2:3 cluster2 2021-04-04T09:40:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster2:3 2021-04-04T09:40:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 local_cluster instance 2021-04-04T09:40:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:40:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:40:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster2 2021-04-04T09:40:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster2 2021-04-04T09:40:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:40:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 admin socket 2021-04-04T09:40:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:40:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.asok 2021-04-04T09:40:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster2:3 2021-04-04T09:40:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2:3 2021-04-04T09:40:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster2:3 cluster instance 2021-04-04T09:40:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster2:3 2021-04-04T09:40:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster2:3 ] 2021-04-04T09:40:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster2 2021-04-04T09:40:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster2 2021-04-04T09:40:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:40:28.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:28.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster2 --name client.mirror.3 pid file 2021-04-04T09:40:28.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2021-04-04T09:40:28.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2021-04-04T09:40:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_pool_mirror_mode cluster2 mirror image 2021-04-04T09:40:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:28.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=image 2021-04-04T09:40:28.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2021-04-04T09:40:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:note: changing mirroring mode from pool to image 2021-04-04T09:40:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ disable_mirror cluster2 mirror test 2021-04-04T09:40:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/test 2021-04-04T09:40:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring disabled 2021-04-04T09:40:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -z ] 2021-04-04T09:40:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test present 2021-04-04T09:40:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:40:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:40:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:40:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:40:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test 2021-04-04T09:40:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test 2021-04-04T09:40:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:40:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id=10e78a1f504 2021-04-04T09:40:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 10e78a1f504 2021-04-04T09:40:29.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -z -o = 10e78a1f504 2021-04-04T09:40:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=present 2021-04-04T09:40:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = present 2021-04-04T09:40:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirrors cluster1 2021-04-04T09:40:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:40:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster1:0 2021-04-04T09:40:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:40:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:40:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:40:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster1:1 2021-04-04T09:40:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:40:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:40:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:40:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:40:29.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:40:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster1:2 2021-04-04T09:40:29.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:40:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:40:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:40:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:40:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd-mirror --cluster cluster1 --id mirror.2 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:40:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ start_mirror cluster1:3 2021-04-04T09:40:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:40:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:40:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:40:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:40:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:40:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:29.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd-mirror --cluster cluster1 --id mirror.3 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:40:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror test deleted 2021-04-04T09:40:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=deleted 2021-04-04T09:40:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:40:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:40:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test 2021-04-04T09:40:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:29.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test 2021-04-04T09:40:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:40:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id=10e78a1f504 2021-04-04T09:40:29.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:40:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test deleted 10e78a1f504 2021-04-04T09:40:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:29.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=deleted 2021-04-04T09:40:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id=10e78a1f504 2021-04-04T09:40:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:40:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:40:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test 2021-04-04T09:40:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test 2021-04-04T09:40:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:40:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id=10e78a1f504 2021-04-04T09:40:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 10e78a1f504 2021-04-04T09:40:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -z 10e78a1f504 -o 10e78a1f504 = 10e78a1f504 2021-04-04T09:40:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=present 2021-04-04T09:40:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test deleted = present 2021-04-04T09:40:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test deleted 10e78a1f504 2021-04-04T09:40:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:30.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=deleted 2021-04-04T09:40:30.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id=10e78a1f504 2021-04-04T09:40:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:40:30.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:40:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test 2021-04-04T09:40:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:30.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:30.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test 2021-04-04T09:40:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:40:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id=10e78a1f504 2021-04-04T09:40:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 10e78a1f504 2021-04-04T09:40:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -z 10e78a1f504 -o 10e78a1f504 = 10e78a1f504 2021-04-04T09:40:30.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=present 2021-04-04T09:40:30.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test deleted = present 2021-04-04T09:40:30.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:40:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test deleted 10e78a1f504 2021-04-04T09:40:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=deleted 2021-04-04T09:40:32.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id=10e78a1f504 2021-04-04T09:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test 2021-04-04T09:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test 2021-04-04T09:40:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:40:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id=10e78a1f504 2021-04-04T09:40:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 10e78a1f504 2021-04-04T09:40:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -z 10e78a1f504 -o 10e78a1f504 = 10e78a1f504 2021-04-04T09:40:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=present 2021-04-04T09:40:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test deleted = present 2021-04-04T09:40:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 4 2021-04-04T09:40:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test deleted 10e78a1f504 2021-04-04T09:40:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=deleted 2021-04-04T09:40:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id=10e78a1f504 2021-04-04T09:40:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:40:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:40:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test 2021-04-04T09:40:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test 2021-04-04T09:40:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:40:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test: (2) No such file or directory 2021-04-04T09:40:36.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:40:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test deleted = deleted 2021-04-04T09:40:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:36.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_pool_mirror_mode cluster2 mirror pool 2021-04-04T09:40:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=pool 2021-04-04T09:40:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror pool 2021-04-04T09:40:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:note: changing mirroring mode from image to pool 2021-04-04T09:40:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_journaling cluster2 mirror test 2021-04-04T09:40:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 feature enable mirror/test journaling 2021-04-04T09:40:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror test present 2021-04-04T09:40:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:36.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=present 2021-04-04T09:40:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:40:36.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:40:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test 2021-04-04T09:40:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test 2021-04-04T09:40:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:40:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test: (2) No such file or directory 2021-04-04T09:40:36.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id= 2021-04-04T09:40:36.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:40:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test present 2021-04-04T09:40:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:40:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:40:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:40:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:40:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test 2021-04-04T09:40:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test 2021-04-04T09:40:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:40:37.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test: (2) No such file or directory 2021-04-04T09:40:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:40:37.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = deleted 2021-04-04T09:40:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test present 2021-04-04T09:40:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:40:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:40:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:40:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:40:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test 2021-04-04T09:40:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test 2021-04-04T09:40:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:40:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test: (2) No such file or directory 2021-04-04T09:40:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:40:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = deleted 2021-04-04T09:40:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:40:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test present 2021-04-04T09:40:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:40:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:40:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:40:40.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:40:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test 2021-04-04T09:40:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:40.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test 2021-04-04T09:40:40.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:40:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test: (2) No such file or directory 2021-04-04T09:40:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:40:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = deleted 2021-04-04T09:40:40.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 4 2021-04-04T09:40:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror test present 2021-04-04T09:40:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=present 2021-04-04T09:40:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:40:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:40:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:40:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror test 2021-04-04T09:40:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/test 2021-04-04T09:40:44.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:40:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id=135d95a808e2 2021-04-04T09:40:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 135d95a808e2 2021-04-04T09:40:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -z -o = 135d95a808e2 2021-04-04T09:40:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=present 2021-04-04T09:40:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test present = present 2021-04-04T09:40:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror test 2021-04-04T09:40:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2021-04-04T09:40:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:40:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:40:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror test started 2021-04-04T09:40:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror 2021-04-04T09:40:45.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:40:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:40:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:40:45.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2021-04-04T09:40:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:40:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:40:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:45.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:40:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:40:45.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:40:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:40:45.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:40:45.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:40:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:40:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:40:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:45.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:45.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:40:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:40:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:40:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:45.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:40:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:40:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 4 closest matches: 2021-04-04T09:40:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:40:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test2 2021-04-04T09:40:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test_clone_v1 2021-04-04T09:40:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test_force_promote 2021-04-04T09:40:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:40:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:40:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:40:45.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:40:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:40:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:40:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:40:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:40:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:40:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:40:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:40:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:40:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:40:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:40:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:40:45.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:40:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 4 closest matches: 2021-04-04T09:40:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test4 2021-04-04T09:40:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test_clone 2021-04-04T09:40:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test_clone1 2021-04-04T09:40:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test_data_pool 2021-04-04T09:40:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:40:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-04-04T09:40:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:40:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:40:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:40:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:40:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:40:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:40:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:45.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:40:45.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:40:45.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:45.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:45.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:45.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:45.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:40:45.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:45.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:40:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:40:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:40:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:40:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-04-04T09:40:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:40:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:40:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:40:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2021-04-04T09:40:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:40:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:40:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:40:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:40:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:40:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:40:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:40:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:40:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:40:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:45.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:40:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:40:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:40:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:40:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:40:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:40:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2021-04-04T09:40:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/test'] 2021-04-04T09:40:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:40:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:40:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:40:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:40:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2021-04-04T09:40:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:40:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:40:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:40:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:40:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:40:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:40:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2021-04-04T09:40:46.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 4 closest matches: 2021-04-04T09:40:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test1 2021-04-04T09:40:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test2 2021-04-04T09:40:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test_clone_v1 2021-04-04T09:40:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test_force_promote 2021-04-04T09:40:46.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:40:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2021-04-04T09:40:46.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:40:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:40:46.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:40:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:46.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:40:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:46.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:40:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:40:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:40:46.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:40:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:40:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:40:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:40:46.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:40:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:40:46.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2021-04-04T09:40:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:no valid command found; 4 closest matches: 2021-04-04T09:40:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test4 2021-04-04T09:40:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test_clone 2021-04-04T09:40:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test_clone1 2021-04-04T09:40:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status mirror/test_data_pool 2021-04-04T09:40:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:40:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2021-04-04T09:40:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:40:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:40:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:40:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:40:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:40:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:40:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:46.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:40:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:40:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:40:46.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:46.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:46.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:46.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:40:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:40:46.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:40:46.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:40:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2021-04-04T09:40:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:40:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:40:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:40:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:40:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:40:47.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:40:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog TEST: non-default namespace image mirroring 2021-04-04T09:40:47.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:40:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:40:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:40:47 TEST: non-default namespace image mirroring 2021-04-04T09:40:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:40:47 TEST: non-default namespace image mirroring 2021-04-04T09:40:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog - replay 2021-04-04T09:40:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + date +%F %T 2021-04-04T09:40:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:40:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:40:47 - replay 2021-04-04T09:40:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:40:47 - replay 2021-04-04T09:40:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror/ns1 test 2021-04-04T09:40:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:47.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2021-04-04T09:40:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:40:47.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/ns1/test 2021-04-04T09:40:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror/ns2 test 2021-04-04T09:40:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2021-04-04T09:40:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:40:47.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling mirror/ns2/test 2021-04-04T09:40:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ enable_mirror cluster2 mirror/ns2 test journal 2021-04-04T09:40:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local mode=journal 2021-04-04T09:40:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/ns2/test journal 2021-04-04T09:40:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring enabled 2021-04-04T09:40:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror/ns1 test 2021-04-04T09:40:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror/ns1 test started 2021-04-04T09:40:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:47.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:40:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:40:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:48.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror/ns1 test started 2021-04-04T09:40:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:40:48.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:40:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:40:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/ns1/test 2021-04-04T09:40:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local+ cluster_instance=cluster1 2021-04-04T09:40:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i+ statelocal 2021-04-04T09:40:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: cluster=cluster1 2021-04-04T09:40:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:40:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:40:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:40:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:40:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:40:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2021-04-04T09:40:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:40:48.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:40:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:48.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:48.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:40:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:40:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:48.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:40:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:48.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:48.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:48.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:48.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:48.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:48.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:48.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:40:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2021-04-04T09:40:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2021-04-04T09:40:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:40:48.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:40:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test 2021-04-04T09:40:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:40:48.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:40:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:40:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:40:48.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:48.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:40:48.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:40:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:40:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:40:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:48.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:40:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:48.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:40:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:40:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:40:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:40:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test 2021-04-04T09:40:48.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2021-04-04T09:40:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:40:48.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:40:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/ns1/test 2021-04-04T09:40:48.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:40:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:40:48.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:40:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:48.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:48.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:40:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:48.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:48.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:40:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:40:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:40:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:40:48.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:40:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:40:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:40:48.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:40:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:40:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns1/test 2021-04-04T09:40:48.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2021-04-04T09:40:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:40:48.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:40:48.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/ns1/test 2021-04-04T09:40:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:40:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:40:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:40:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:48.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:40:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:48.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:40:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:40:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:40:48.647 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:40:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:40:48.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:40:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:40:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:40:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:40:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:40:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:40:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns1/test 2021-04-04T09:40:48.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2021-04-04T09:40:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:40:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:40:48.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:49.771 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:40:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:40:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2021-04-04T09:40:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:40:49.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:40:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:40:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:40:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:40:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:49.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:49.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:40:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2021-04-04T09:40:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2021-04-04T09:40:49.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:40:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:40:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test 2021-04-04T09:40:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:40:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:40:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:40:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:40:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:40:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:40:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:40:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:40:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:40:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:40:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:40:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:40:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:40:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:40:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:40:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test 2021-04-04T09:40:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2021-04-04T09:40:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:40:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:40:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/ns1/test 2021-04-04T09:40:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:40:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:40:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:40:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:40:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:40:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:40:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:40:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:40:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:40:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:40:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:40:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:40:50.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:40:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:40:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:40:50.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns1/test 2021-04-04T09:40:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2021-04-04T09:40:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:40:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:40:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/ns1/test 2021-04-04T09:40:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:40:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:50.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:40:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:40:50.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:50.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:50.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:40:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:40:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:50.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:40:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:40:50.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:40:50.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:40:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:50.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:40:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:40:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:40:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:40:50.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:40:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:40:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:40:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns1/test 2021-04-04T09:40:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test'] 2021-04-04T09:40:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:40:50.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:40:50.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:40:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:40:52.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:40:52.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test 2021-04-04T09:40:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:40:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:40:52.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:52.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:52.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:52.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:52.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:52.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:52.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:52.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:52.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:40:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:40:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:40:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:52.284 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:52.285 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:52.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:40:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test 2021-04-04T09:40:52.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:40:52.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + echo "state": "Replaying" 2021-04-04T09:40:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i Replaying 2021-04-04T09:40:52.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:40:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:40:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:40:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror/ns2 test 2021-04-04T09:40:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:52.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror/ns2 test started 2021-04-04T09:40:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:40:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:40:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror/ns2 test started 2021-04-04T09:40:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:40:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:40:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:40:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/ns2/test 2021-04-04T09:40:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local+ cluster_instance=cluster1 2021-04-04T09:40:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:40:53.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:40:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:40:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:40:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:40:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:40:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/ns2/test 2021-04-04T09:40:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:40:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:40:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:40:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:40:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:40:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:53.427 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:53.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:40:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns2/test 2021-04-04T09:40:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:40:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:40:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:40:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:40:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:40:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:40:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:53.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror/ns1 test 100 2021-04-04T09:40:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:40:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:40:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:40:53.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/ns1/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:40:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:40:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:40:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2500.06 bytes/sec: 9.8 MiB/s 2021-04-04T09:40:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror/ns2 test 100 2021-04-04T09:40:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:40:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/ns2/test --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:40:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:40:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:40:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2500.06 bytes/sec: 9.8 MiB/s 2021-04-04T09:40:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns1 test 2021-04-04T09:40:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:40:53.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:40:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns1 test 2021-04-04T09:40:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:40:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:40:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:40:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:40:53.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:40:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror/ns1 test 2021-04-04T09:40:53.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:53.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:53.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:40:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:40:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/ns1/test 2021-04-04T09:40:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test 2021-04-04T09:40:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:40:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:40:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:40:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:40:54.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:40:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:40:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test 2021-04-04T09:40:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:40:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:40:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:54.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:40:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:40:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:40:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:54.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:40:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test 2021-04-04T09:40:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror/ns1 test 2021-04-04T09:40:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:54.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test 2021-04-04T09:40:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:54.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:40:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test.status 2021-04-04T09:40:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test.status 2021-04-04T09:40:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:40:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test.status 2021-04-04T09:40:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test 2021-04-04T09:40:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test.status 2021-04-04T09:40:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:40:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:40:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:40:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:40:54.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:40:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test.status 2021-04-04T09:40:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:40:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror/ns1 test 2021-04-04T09:40:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test ..* 2021-04-04T09:40:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:40:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test.status 2021-04-04T09:40:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test.status 2021-04-04T09:40:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:40:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test.status 2021-04-04T09:40:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test 2021-04-04T09:40:54.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test.status 2021-04-04T09:40:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:40:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:40:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:40:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:40:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:40:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test.status 2021-04-04T09:40:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:40:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=2, entry_tid=101 -a object_number=1, tag_tid=2, entry_tid=101 = object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:40:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:54.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns2 test 2021-04-04T09:40:54.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:40:54.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:40:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns2 test 2021-04-04T09:40:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:40:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:40:54.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:40:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:40:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:40:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror/ns2 test 2021-04-04T09:40:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:40:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test ] 2021-04-04T09:40:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/ns2/test 2021-04-04T09:40:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns2/test 2021-04-04T09:40:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:40:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:40:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:40:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:40:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:40:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:40:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns2/test 2021-04-04T09:40:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:40:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:40:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:40:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:40:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:40:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:40:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:40:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:40:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:40:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:40:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:40:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:40:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:40:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:40:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:40:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:40:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:40:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:40:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:40:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns2/test 2021-04-04T09:40:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror/ns2 test 2021-04-04T09:40:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns2 test 2021-04-04T09:40:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:40:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns2-test.status 2021-04-04T09:40:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns2-test.status 2021-04-04T09:40:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:40:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns2-test.status 2021-04-04T09:40:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns2-test.status 2021-04-04T09:40:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd --cluster cluster2 journal status --image mirror/ns2/test 2021-04-04T09:40:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:40:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:40:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:40:54.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:40:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:40:54.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns2-test.status 2021-04-04T09:40:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:40:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror/ns2 test 2021-04-04T09:40:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns2 test ..* 2021-04-04T09:40:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:40:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns2-test.status 2021-04-04T09:40:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns2-test.status 2021-04-04T09:40:54.879 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:40:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns2-test.status 2021-04-04T09:40:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns2/test 2021-04-04T09:40:54.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns2-test.status 2021-04-04T09:40:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:40:54.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:40:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:40:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:40:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:40:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns2-test.status 2021-04-04T09:40:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:40:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=2, entry_tid=101 -a object_number=1, tag_tid=2, entry_tid=101 = object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:40:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns1 test up+replaying primary_position 2021-04-04T09:40:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:40:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:40:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:40:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror/ns1 test up+replaying primary_position 2021-04-04T09:40:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:40:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:40:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:40:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror/ns1-test.mirror_status 2021-04-04T09:40:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror/ns1-test.mirror_status 2021-04-04T09:40:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:40:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns1-test.mirror_status 2021-04-04T09:40:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/ns1/test 2021-04-04T09:40:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns1-test.mirror_status 2021-04-04T09:40:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:40:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: 909bceaa-c260-415a-a60e-eee31d4f3e89 2021-04-04T09:40:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:40:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T09:40:56.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2021-04-04T09:40:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:40:51 2021-04-04T09:40:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:40:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:40:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: down+unknown 2021-04-04T09:40:56.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: status not found 2021-04-04T09:40:56.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04T09:40:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns1-test.mirror_status 2021-04-04T09:40:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:40:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .*primary_position /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns1-test.mirror_status 2021-04-04T09:40:56.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T09:40:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:40:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:40:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:40:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:40:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns1-test.mirror_status 2021-04-04T09:40:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:40:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns1-test.mirror_status 2021-04-04T09:40:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:40:51 2021-04-04T09:40:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns1 test up+replaying 2021-04-04 09:40:51 2021-04-04T09:40:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:40:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:40:51 2021-04-04T09:40:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror/ns1.mirror_status 2021-04-04T09:40:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror/ns1.mirror_status 2021-04-04T09:40:56.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:40:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns1.mirror_status 2021-04-04T09:40:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror/ns1 --verbose --format xml 2021-04-04T09:40:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:40:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:40:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:40:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:40:51 2021-04-04T09:40:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:40:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:40:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:40:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:40:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:40:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_status_in_pool_dir cluster1 mirror/ns2 test up+replaying primary_position 2021-04-04T09:40:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:40:56.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:40:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:40:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:40:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_status_in_pool_dir cluster1 mirror/ns2 test up+replaying primary_position 2021-04-04T09:40:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:40:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local description_pattern=primary_position 2021-04-04T09:40:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local service_pattern= 2021-04-04T09:40:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror/ns2-test.mirror_status 2021-04-04T09:40:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror/ns2-test.mirror_status 2021-04-04T09:40:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:40:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns2-test.mirror_status 2021-04-04T09:40:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/ns2/test 2021-04-04T09:40:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:tee /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns2-test.mirror_status 2021-04-04T09:40:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:test: 2021-04-04T09:40:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: global_id: af2694d6-07b0-4e37-95e1-92663262de9a 2021-04-04T09:40:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: up+replaying 2021-04-04T09:40:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T09:40:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: service: mirror.0 on smithi122 2021-04-04T09:40:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04 09:40:51 2021-04-04T09:40:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: peer_sites: 2021-04-04T09:40:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: name: cluster2 2021-04-04T09:40:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: state: down+unknown 2021-04-04T09:40:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: description: status not found 2021-04-04T09:40:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: last_update: 2021-04-04T09:40:57.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns2-test.mirror_status 2021-04-04T09:40:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:40:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^ description: .*primary_position /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns2-test.mirror_status 2021-04-04T09:40:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":3,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T09:40:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n ] 2021-04-04T09:40:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:40:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep ^up+ 2021-04-04T09:40:57.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:40:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep service: *mirror..* on /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns2-test.mirror_status 2021-04-04T09:40:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:40:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns2-test.mirror_status 2021-04-04T09:40:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update=2021-04-04 09:40:51 2021-04-04T09:40:57.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_mirror_pool_status_verbose cluster1 mirror/ns2 test up+replaying 2021-04-04 09:40:51 2021-04-04T09:40:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state_pattern=up+replaying 2021-04-04T09:40:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local prev_last_update=2021-04-04 09:40:51 2021-04-04T09:40:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror/ns2.mirror_status 2021-04-04T09:40:57.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror/ns2.mirror_status 2021-04-04T09:40:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:40:57.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns2.mirror_status 2021-04-04T09:40:57.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 mirror pool status mirror/ns2 --verbose --format xml 2021-04-04T09:40:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: failed to query services: (13) Permission denied 2021-04-04T09:40:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local last_update state 2021-04-04T09:40:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2021-04-04T09:40:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ last_update=2021-04-04 09:40:51 2021-04-04T09:40:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2021-04-04T09:40:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ state=up+replaying 2021-04-04T09:40:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo up+replaying 2021-04-04T09:40:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep up+replaying 2021-04-04T09:40:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:up+replaying 2021-04-04T09:40:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_images mirror/ns1 test 2021-04-04T09:40:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2021-04-04T09:40:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test.export 2021-04-04T09:40:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test.export 2021-04-04T09:40:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:40:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test.export 2021-04-04T09:40:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror/ns1-test.export 2021-04-04T09:40:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror/ns1-test.export 2021-04-04T09:40:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:40:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns1-test.export 2021-04-04T09:40:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns1-test.export 2021-04-04T09:40:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 export mirror/ns1/test /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test.export 2021-04-04T09:40:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:40:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 export mirror/ns1/test /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns1-test.export 2021-04-04T09:40:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:40:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns1-test.export 2021-04-04T09:40:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns1-test.export 2021-04-04T09:40:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ compare_images mirror/ns2 test 2021-04-04T09:40:57.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:57.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:57.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret=0 2021-04-04T09:40:57.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns2-test.export 2021-04-04T09:40:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns2-test.export 2021-04-04T09:40:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:40:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local rmt_export=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns2-test.export 2021-04-04T09:40:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster1-mirror/ns2-test.export 2021-04-04T09:40:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1-mirror/ns2-test.export 2021-04-04T09:40:57.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:40:57.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loc_export=/tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns2-test.export 2021-04-04T09:40:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns2-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns2-test.export 2021-04-04T09:40:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 export mirror/ns2/test /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns2-test.export 2021-04-04T09:40:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:40:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 export mirror/ns2/test /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns2-test.export 2021-04-04T09:40:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2021-04-04T09:40:58.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmp /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns2-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns2-test.export 2021-04-04T09:40:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rm -f /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns2-test.export /tmp/tmp.zj6T3cKcnO/cluster1-mirror_ns2-test.export 2021-04-04T09:40:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog - disable mirroring / delete image 2021-04-04T09:40:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:40:58.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:40:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:40:58 - disable mirroring / delete image 2021-04-04T09:40:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:40:58 - disable mirroring / delete image 2021-04-04T09:40:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_image_retry cluster2 mirror/ns1 test 2021-04-04T09:40:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:58.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:40:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ remove_image cluster2 mirror/ns1 test 2021-04-04T09:40:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 snap purge mirror/ns1/test 2021-04-04T09:40:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 rm mirror/ns1/test 2021-04-04T09:40:58.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2021-04-04T09:40:58.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:58.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ disable_mirror cluster2 mirror/ns2 test 2021-04-04T09:40:58.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:58.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/ns2/test 2021-04-04T09:40:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirroring disabled 2021-04-04T09:40:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror/ns1 test deleted 2021-04-04T09:40:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=deleted 2021-04-04T09:40:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:40:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:40:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror/ns1 test 2021-04-04T09:40:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/ns1/test 2021-04-04T09:40:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:40:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test: (2) No such file or directory 2021-04-04T09:40:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id= 2021-04-04T09:40:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:40:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror/ns1 test deleted 2021-04-04T09:40:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=deleted 2021-04-04T09:40:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:40:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:40:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:40:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror/ns1 test 2021-04-04T09:40:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/ns1/test 2021-04-04T09:40:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:40:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test: (2) No such file or directory 2021-04-04T09:40:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:40:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test deleted = deleted 2021-04-04T09:40:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_present cluster1 mirror/ns2 test deleted 2021-04-04T09:40:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=deleted 2021-04-04T09:40:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:40:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:40:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror/ns2 test 2021-04-04T09:40:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/ns2/test 2021-04-04T09:40:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:40:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test: (2) No such file or directory 2021-04-04T09:40:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ image_id= 2021-04-04T09:40:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.1 2021-04-04T09:40:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_present cluster1 mirror/ns2 test deleted 2021-04-04T09:40:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=deleted 2021-04-04T09:40:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image_id= 2021-04-04T09:40:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=deleted 2021-04-04T09:40:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_image_id 2021-04-04T09:40:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_id cluster1 mirror/ns2 test 2021-04-04T09:40:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns2 2021-04-04T09:40:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test 2021-04-04T09:40:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster=cluster1 info mirror/ns2/test 2021-04-04T09:40:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2021-04-04T09:40:58.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: error opening image test: (2) No such file or directory 2021-04-04T09:40:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_image_id= 2021-04-04T09:40:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:40:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test deleted = deleted 2021-04-04T09:40:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:40:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ testlog - data pool 2021-04-04T09:40:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee -a /tmp/tmp.zj6T3cKcnO/rbd-mirror.test.log 2021-04-04T09:40:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ date +%F %T 2021-04-04T09:40:58.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04 09:40:58 - data pool 2021-04-04T09:40:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:2021-04-04 09:40:58 - data pool 2021-04-04T09:40:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ dp_image=test_data_pool 2021-04-04T09:40:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ create_image cluster2 mirror/ns1 test_data_pool 128 --data-pool mirror_parent 2021-04-04T09:40:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:40:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size=128 2021-04-04T09:40:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n 128 ] 2021-04-04T09:40:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=128 2021-04-04T09:40:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:40:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 create --size 128 --image-feature layering,exclusive-lock,journaling --data-pool mirror_parent mirror/ns1/test_data_pool 2021-04-04T09:40:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_data_pool cluster2 mirror/ns1 test_data_pool 2021-04-04T09:40:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:40:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:40:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 info mirror/ns1/test_data_pool 2021-04-04T09:40:59.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk $1 == "data_pool:" {print $2} 2021-04-04T09:40:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ data_pool=mirror_parent 2021-04-04T09:40:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test mirror_parent = mirror_parent 2021-04-04T09:40:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_started cluster1 mirror/ns1 test_data_pool 2021-04-04T09:40:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:40:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_image_replay_state cluster1 mirror/ns1 test_data_pool started 2021-04-04T09:40:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:40:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:40:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:40:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local state=started 2021-04-04T09:40:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s 2021-04-04T09:40:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:41:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test_image_replay_state cluster1 mirror/ns1 test_data_pool started 2021-04-04T09:41:00.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local test_state=started 2021-04-04T09:41:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_result 2021-04-04T09:41:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local current_state=stopped 2021-04-04T09:41:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + local cluster_instance=cluster1 2021-04-04T09:41:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:00.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:grep -i state 2021-04-04T09:41:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:41:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:41:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:41:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:00.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:41:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:41:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:41:00.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:00.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:00.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:41:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:00.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:00.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:41:00.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:00.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:00.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:41:00.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:00.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:00.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:41:00.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:00.243 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:41:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:41:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:00.244 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:41:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:41:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:41:00.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:41:00.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:41:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:41:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:41:00.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:41:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:41:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:41:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:41:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:41:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:41:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:41:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:41:00.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:41:00.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:00.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:00.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:41:00.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:41:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:41:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:41:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:41:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:41:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:41:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:41:00.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:41:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:41:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:41:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:41:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:41:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:41:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:41:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:41:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:41:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:41:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:00.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:41:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:41:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:00.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:41:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:41:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:41:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:41:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:41:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:41:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:41:00.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:00.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1 2021-04-04T09:41:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:41:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:41:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:41:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:41:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:01.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:41:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:41:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:41:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:41:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:01.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:41:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:41:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:41:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:41:01.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:41:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:41:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:01.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:41:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:41:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:2 2021-04-04T09:41:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 cluster instance 2021-04-04T09:41:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:41:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:41:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:41:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:01.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:01.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:41:01.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:41:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:2 cluster1 2021-04-04T09:41:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:2 2021-04-04T09:41:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:01.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2021-04-04T09:41:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:2 2021-04-04T09:41:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:41:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 2 = cluster1:2 ] 2021-04-04T09:41:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=2 2021-04-04T09:41:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=2 2021-04-04T09:41:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2021-04-04T09:41:01.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:41:01.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:41:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2021-04-04T09:41:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:41:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:3 2021-04-04T09:41:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 cluster instance 2021-04-04T09:41:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:41:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:01.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:41:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:41:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:01.966 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:01.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:41:01.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:41:01.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:3 cluster1 2021-04-04T09:41:01.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:3 2021-04-04T09:41:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:01.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2021-04-04T09:41:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:3 2021-04-04T09:41:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:01.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:41:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 3 = cluster1:3 ] 2021-04-04T09:41:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=3 2021-04-04T09:41:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=3 2021-04-04T09:41:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2021-04-04T09:41:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:41:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:41:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2021-04-04T09:41:01.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:41:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:41:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:41:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:41:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:41:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:04.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:41:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:41:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:41:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:41:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:04.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:41:04.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror status : get status for rbd mirror 2021-04-04T09:41:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:41:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:41:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:41:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:41:04.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:41:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:04.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:41:04.248 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:04.249 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:04.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/ns1/test_data_pool 2021-04-04T09:41:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:41:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status_result= "state": "Replaying" 2021-04-04T09:41:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo "state": "Replaying" 2021-04-04T09:41:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ grep -i Replaying 2021-04-04T09:41:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:41:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ current_state=started 2021-04-04T09:41:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test started = started 2021-04-04T09:41:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:41:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_image_data_pool cluster1 mirror/ns1 test_data_pool 2021-04-04T09:41:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:04.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:04.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + rbd --cluster cluster1 info mirror/ns1/test_data_pool 2021-04-04T09:41:04.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:awk $1 == "data_pool:" {print $2} 2021-04-04T09:41:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ data_pool=mirror_parent 2021-04-04T09:41:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test mirror_parent = mirror_parent 2021-04-04T09:41:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ write_image cluster2 mirror/ns1 test_data_pool 100 2021-04-04T09:41:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:04.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local count=100 2021-04-04T09:41:04.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local size= 2021-04-04T09:41:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n 2021-04-04T09:41:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ size=4096 2021-04-04T09:41:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 bench mirror/ns1/test_data_pool --io-type write --io-size 4096 --io-threads 1 --io-total 409600 --io-pattern rand 2021-04-04T09:41:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:bench type write io_size 4096 io_threads 1 bytes 409600 pattern random 2021-04-04T09:41:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: SEC OPS OPS/SEC BYTES/SEC 2021-04-04T09:41:04.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:elapsed: 0 ops: 100 ops/sec: 2500.06 bytes/sec: 9.8 MiB/s 2021-04-04T09:41:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:41:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:04.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ journal = journal ] 2021-04-04T09:41:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ wait_for_journal_replay_complete cluster1 cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1 2021-04-04T09:41:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:04.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local s master_pos mirror_pos last_mirror_pos 2021-04-04T09:41:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local master_tag master_entry mirror_tag mirror_entry 2021-04-04T09:41:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ true 2021-04-04T09:41:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.2 2021-04-04T09:41:04.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2021-04-04T09:41:04.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:04.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:41:04.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:41:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:41:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:41:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:41:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:41:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:41:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:41:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:41:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:41:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:41:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:41:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:41:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:04.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:04.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:41:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:04.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:41:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:41:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:41:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:04.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:04.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:41:04.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:41:04.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:41:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:04.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:04.891 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:04.892 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:04.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:41:04.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:04.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:41:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:05.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:41:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:05.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:05.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:41:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool ..* 2021-04-04T09:41:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:41:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:05.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos= 2021-04-04T09:41:05.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=2, entry_tid=101 -a object_number=1, tag_tid=2, entry_tid=101 = 2021-04-04T09:41:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test != 2021-04-04T09:41:05.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.4 2021-04-04T09:41:05.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2021-04-04T09:41:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:05.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:41:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:41:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:41:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:41:05.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:41:05.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:41:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:41:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:41:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:41:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:05.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:05.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:41:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:05.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:05.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:41:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:41:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:41:05.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:05.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:05.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:41:05.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:41:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:41:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:41:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:41:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:41:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:05.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:41:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:41:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:05.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:41:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:41:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:05.893 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:05.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:41:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool ..* 2021-04-04T09:41:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:05.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:41:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + rbd --cluster cluster2 journal status --imagetee mirror/ns1/test_data_pool /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: 2021-04-04T09:41:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:05.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos= 2021-04-04T09:41:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=2, entry_tid=101 -a object_number=1, tag_tid=2, entry_tid=101 = 2021-04-04T09:41:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test != 2021-04-04T09:41:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0.8 2021-04-04T09:41:06.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2021-04-04T09:41:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:06.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:41:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:41:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:41:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:41:06.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:41:06.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:41:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:41:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:41:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:41:06.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:41:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:06.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:41:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:41:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:06.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:41:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:41:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:06.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:06.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:06.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:41:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:41:06.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:06.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:41:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:06.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:06.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:06.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:41:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:41:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:41:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:06.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:06.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:41:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:41:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:07.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:07.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:07.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:41:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:07.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:07.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:07.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:07.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:07.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:41:07.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:07.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool ..* 2021-04-04T09:41:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:41:07.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:07.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:07.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:07.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:07.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:07.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos= 2021-04-04T09:41:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=2, entry_tid=101 -a object_number=1, tag_tid=2, entry_tid=101 = 2021-04-04T09:41:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test != 2021-04-04T09:41:07.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 1.6 2021-04-04T09:41:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2021-04-04T09:41:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:41:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:41:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:41:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:41:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:41:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:41:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:41:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:41:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:08.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:41:08.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:08.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:08.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:41:08.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:08.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:08.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:41:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:41:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:41:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:08.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:08.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:41:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:08.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:41:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:41:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:41:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:08.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:08.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:41:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:41:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:08.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:41:08.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:08.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:08.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:08.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:41:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:08.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:41:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:09.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:09.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:41:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:09.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:41:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool ..* 2021-04-04T09:41:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:41:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + echosed -e s|[/ ]|_|g 2021-04-04T09:41:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:09.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos= 2021-04-04T09:41:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=2, entry_tid=101 -a object_number=1, tag_tid=2, entry_tid=101 = 2021-04-04T09:41:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test != 2021-04-04T09:41:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:41:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2021-04-04T09:41:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:11.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:41:11.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:41:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:41:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:41:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:41:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:41:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:41:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:41:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:41:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:41:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:11.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:11.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:41:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:41:11.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:41:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:11.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:41:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:11.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:41:11.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:41:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:41:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:41:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:41:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:41:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:11.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:41:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:41:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:41:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:11.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:11.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:11.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:11.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:41:11.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:11.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:11.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:11.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:11.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool ..* 2021-04-04T09:41:11.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:41:11.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:11.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:11.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:11.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:11.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos= 2021-04-04T09:41:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=2, entry_tid=101 -a object_number=1, tag_tid=2, entry_tid=101 = 2021-04-04T09:41:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test != 2021-04-04T09:41:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 2 2021-04-04T09:41:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2021-04-04T09:41:13.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:41:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:41:13.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:41:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:41:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:41:13.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:41:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:41:13.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:41:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:41:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:13.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:41:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:41:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:41:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:41:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:13.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:41:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:41:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:41:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:41:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:41:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:41:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:41:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:41:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:13.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:41:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:13.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:41:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:13.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:13.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:41:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:13.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:13.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool ..* 2021-04-04T09:41:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:13.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:13.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:41:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:13.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:13.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:13.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos= 2021-04-04T09:41:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=2, entry_tid=101 -a object_number=1, tag_tid=2, entry_tid=101 = 2021-04-04T09:41:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test != 2021-04-04T09:41:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 4 2021-04-04T09:41:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2021-04-04T09:41:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:17.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:41:17.986 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:41:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:41:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:41:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:41:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:41:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:41:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:41:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:41:17.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:17.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:17.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:41:17.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:17.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:17.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:41:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:41:18.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:41:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:41:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:18.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:18.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:41:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:41:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:41:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:41:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:41:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:18.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:41:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:18.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:18.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:41:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:18.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:41:18.250 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:18.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:18.251 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:18.253 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:41:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:18.256 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:18.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:18.257 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:18.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:18.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:41:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool ..* 2021-04-04T09:41:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:18.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:41:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:18.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:18.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:18.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos= 2021-04-04T09:41:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=2, entry_tid=101 -a object_number=1, tag_tid=2, entry_tid=101 = 2021-04-04T09:41:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test != 2021-04-04T09:41:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 4 2021-04-04T09:41:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2021-04-04T09:41:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:41:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:41:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:22.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:22.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:41:22.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:41:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:41:22.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:41:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:41:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:41:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:41:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:41:22.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:22.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:22.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:22.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:22.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:22.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:41:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:41:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:22.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:41:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:22.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:22.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:41:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:22.416 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:22.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:41:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:41:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:41:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:41:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:41:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:41:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:41:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:41:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:22.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:22.651 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:41:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:22.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:22.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:41:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool ..* 2021-04-04T09:41:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:41:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image+ mirror/ns1/test_data_pool 2021-04-04T09:41:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:22.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:22.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos= 2021-04-04T09:41:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=2, entry_tid=101 -a object_number=1, tag_tid=2, entry_tid=101 = 2021-04-04T09:41:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test != 2021-04-04T09:41:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:41:30.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2021-04-04T09:41:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:41:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:41:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:30.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:41:30.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:41:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:41:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:41:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:41:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:41:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:41:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:41:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:41:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:41:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:41:30.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:30.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:41:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:30.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:41:30.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:30.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:41:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:41:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:30.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:41:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:41:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:41:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:30.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:30.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:41:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:30.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:31.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:41:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:31.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:41:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:31.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:31.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:41:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:31.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool ..* 2021-04-04T09:41:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:31.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:41:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:31.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:31.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:31.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:31.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos= 2021-04-04T09:41:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=2, entry_tid=101 -a object_number=1, tag_tid=2, entry_tid=101 = 2021-04-04T09:41:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test != 2021-04-04T09:41:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 8 2021-04-04T09:41:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2021-04-04T09:41:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:41:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:41:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:39.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:41:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:39.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:41:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:41:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:41:39.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:41:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:41:39.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:41:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:39.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:41:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:39.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:41:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:41:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:41:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:39.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:39.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:41:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:41:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:41:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:39.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:41:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:41:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:39.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:41:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:39.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:39.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:41:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:39.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:39.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:41:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:39.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:39.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:41:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:39.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:39.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:41:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool ..* 2021-04-04T09:41:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:41:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos= 2021-04-04T09:41:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=2, entry_tid=101 -a object_number=1, tag_tid=2, entry_tid=101 = 2021-04-04T09:41:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test != 2021-04-04T09:41:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 16 2021-04-04T09:41:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2021-04-04T09:41:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:41:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:41:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:41:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:41:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:41:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:41:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:41:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:41:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:41:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:41:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:41:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:41:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:41:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:41:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:41:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:41:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:41:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:41:55.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:55.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:55.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:41:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:41:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:41:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:41:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:41:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:41:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:41:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:41:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:41:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:41:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:41:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:41:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:41:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:41:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:41:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:41:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:41:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:41:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:41:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:41:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:41:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:41:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:41:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:41:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:41:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:41:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:41:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:56.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:56.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:56.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:41:56.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:56.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:56.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:56.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:56.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:56.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:41:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:41:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool ..* 2021-04-04T09:41:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:41:56.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:41:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:41:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:41:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:41:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:41:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:41:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:41:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:41:56.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:41:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:41:56.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:41:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:41:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos= 2021-04-04T09:41:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=2, entry_tid=101 -a object_number=1, tag_tid=2, entry_tid=101 = 2021-04-04T09:41:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test != 2021-04-04T09:41:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 16 2021-04-04T09:42:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2021-04-04T09:42:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:42:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:42:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:42:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:42:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:42:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:42:12.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:42:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:42:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:42:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:42:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:42:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:42:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:42:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:42:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:42:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:42:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:42:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:42:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:42:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:42:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:42:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:42:12.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:42:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:42:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:42:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:42:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:42:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:42:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:42:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:42:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:42:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:42:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:42:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:42:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:42:12.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:42:12.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:42:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:42:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:42:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:42:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:42:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:42:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:42:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:42:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:42:12.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:42:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:42:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:42:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:42:12.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:42:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:42:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:42:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:42:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:42:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:42:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:42:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:42:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:42:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:42:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:42:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:42:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:42:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:42:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:42:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:42:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:42:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:42:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:42:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:42:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:42:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:42:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:42:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:42:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:42:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:42:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:42:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:42:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:42:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:42:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:42:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:42:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:42:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:42:12.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:42:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:42:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:42:12.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:42:12.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:42:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:42:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:42:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:42:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:42:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:42:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:42:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:42:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:42:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:42:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:42:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:42:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:42:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:42:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:42:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:42:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:42:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:42:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:42:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:42:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:42:12.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:42:12.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:42:12.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:42:12.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:42:12.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:42:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:42:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:42:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool ..* 2021-04-04T09:42:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:42:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:42:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:42:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:42:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:42:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:42:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:42:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:42:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool+ 2021-04-04T09:42:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:42:12.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:42:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:42:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:42:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:42:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:42:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:42:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos= 2021-04-04T09:42:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=2, entry_tid=101 -a object_number=1, tag_tid=2, entry_tid=101 = 2021-04-04T09:42:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test != 2021-04-04T09:42:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 32 2021-04-04T09:42:44.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2021-04-04T09:42:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:42:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:42:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:42:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:42:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:42:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:42:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:42:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:42:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:42:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:42:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:42:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:42:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:42:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:42:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:42:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:42:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:42:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:42:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:42:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:42:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:42:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:42:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:42:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:42:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:42:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:42:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:42:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:42:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:42:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:42:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:42:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:42:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:42:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:42:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:42:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:42:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:42:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:42:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:42:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:42:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:42:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:42:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:42:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:42:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:42:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:42:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:42:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:42:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:42:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:42:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:42:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:42:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:42:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:42:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:42:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:42:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:42:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:42:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:42:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:42:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:42:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:42:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:42:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:42:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:42:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:42:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:42:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:42:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:42:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:42:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:42:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:42:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:42:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:42:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:42:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:42:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:42:44.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:42:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:42:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:42:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:42:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:42:44.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:42:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:42:44.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:42:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:42:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:42:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:42:44.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:42:44.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:42:44.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:42:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:42:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:42:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:42:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:42:44.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:42:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:42:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:42:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:42:44.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:42:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:42:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:42:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + teerbd /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:42:44.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:42:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:42:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:42:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:42:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:42:45.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:42:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:42:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:42:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:42:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:42:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:42:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:42:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool ..* 2021-04-04T09:42:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:42:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:42:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:42:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:42:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:42:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:42:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:42:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:42:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:42:45.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:42:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:42:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:42:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:42:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:42:45.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:42:45.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:42:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos= 2021-04-04T09:42:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=2, entry_tid=101 -a object_number=1, tag_tid=2, entry_tid=101 = 2021-04-04T09:42:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test != 2021-04-04T09:42:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 32 2021-04-04T09:43:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ flush cluster1 mirror/ns1 test_data_pool 2021-04-04T09:43:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:43:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:43:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:43:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cmd=rbd mirror flush 2021-04-04T09:43:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -n test_data_pool ] 2021-04-04T09:43:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cmd=rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:43:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemons cluster1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:43:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_instance=cluster1 2021-04-04T09:43:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:43:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:43:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance=cluster1 2021-04-04T09:43:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local loop_instance 2021-04-04T09:43:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sleep 0 2021-04-04T09:43:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ cluster1 != cluster1 ] 2021-04-04T09:43:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:43:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:0 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:43:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:43:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:43:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:43:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:43:17.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:43:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:43:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:43:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:43:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:43:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:43:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:43:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:43:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:43:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:43:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:43:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:43:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:43:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:43:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:43:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:43:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:43:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:43:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:43:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:43:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:43:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:43:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:43:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:43:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:43:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:43:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:43:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:43:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:43:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:43:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Invalid command: unused arguments: ['mirror/ns1/test_data_pool'] 2021-04-04T09:43:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd mirror flush : flush rbd mirror 2021-04-04T09:43:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:admin_socket: invalid command 2021-04-04T09:43:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ admin_daemon cluster1:1 rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:43:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:43:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ shift 2021-04-04T09:43:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:43:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:43:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:43:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:43:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:43:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:43:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:43:17.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:43:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:43:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:43:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:43:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:43:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:43:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:43:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:43:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:43:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:43:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:43:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:43:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:43:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:43:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:43:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:43:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:43:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:43:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:43:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:43:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:43:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:43:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:43:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:43:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror flush mirror/ns1/test_data_pool 2021-04-04T09:43:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 0 2021-04-04T09:43:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_master_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:43:17.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:43:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:43:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:43:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:43:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:43:17.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:43:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:43:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp= 2021-04-04T09:43:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:43:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:43:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:43:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:43:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:43:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ tee /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:43:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:43:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:43:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:43:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:43:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:43:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ master_pos=object_number=1, tag_tid=2, entry_tid=101 2021-04-04T09:43:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_mirror_journal_position cluster2 mirror/ns1 test_data_pool 2021-04-04T09:43:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:43:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:43:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:43:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ get_journal_position cluster2 mirror/ns1 test_data_pool ..* 2021-04-04T09:43:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster2 2021-04-04T09:43:17.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pool=mirror/ns1 2021-04-04T09:43:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local image=test_data_pool 2021-04-04T09:43:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local id_regexp=..* 2021-04-04T09:43:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mkfname cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:43:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2-mirror/ns1-test_data_pool.status 2021-04-04T09:43:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -e s|[/ ]|_|g 2021-04-04T09:43:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local status_log=/tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:43:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ + teerbd /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:43:17.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: --cluster cluster2 journal status --image mirror/ns1/test_data_pool 2021-04-04T09:43:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:minimum_set: 0 2021-04-04T09:43:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:active_set: 0 2021-04-04T09:43:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:registered clients: 2021-04-04T09:43:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:43:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ sed -nEe s/^.*\[id=..*,.*positions=\[\[([^]]*)\],.*state=connected.*$/\1/p /tmp/tmp.zj6T3cKcnO/cluster2-mirror_ns1-test_data_pool.status 2021-04-04T09:43:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ mirror_pos= 2021-04-04T09:43:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test -n object_number=1, tag_tid=2, entry_tid=101 -a object_number=1, tag_tid=2, entry_tid=101 = 2021-04-04T09:43:17.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test != 2021-04-04T09:43:17.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ test = 2021-04-04T09:43:17.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ return 1 2021-04-04T09:43:17.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cleanup 1 2021-04-04T09:43:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local error_code=1 2021-04-04T09:43:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2021-04-04T09:43:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 -ne 0 ] 2021-04-04T09:43:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status 2021-04-04T09:43:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster daemon image_pool image_ns image 2021-04-04T09:43:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 status 2021-04-04T09:43:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 status 2021-04-04T09:43:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster1 -s 2021-04-04T09:43:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: cluster: 2021-04-04T09:43:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 7fae7b04-7e84-4b67-8401-2f7942dda448 2021-04-04T09:43:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: HEALTH_OK 2021-04-04T09:43:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:18.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: services: 2021-04-04T09:43:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mon: 1 daemons, quorum a (age 14m) 2021-04-04T09:43:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mgr: x(active, since 14m) 2021-04-04T09:43:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: osd: 3 osds: 3 up (since 14m), 3 in (since 14m) 2021-04-04T09:43:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: rbd-mirror: 4 daemons active (1 hosts) 2021-04-04T09:43:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: data: 2021-04-04T09:43:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: pools: 4 pools, 137 pgs 2021-04-04T09:43:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: objects: 420 objects, 835 MiB 2021-04-04T09:43:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: usage: 2.6 GiB used, 265 GiB / 268 GiB avail 2021-04-04T09:43:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: pgs: 137 active+clean 2021-04-04T09:43:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: io: 2021-04-04T09:43:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client: 8.8 KiB/s rd, 511 B/s wr, 8 op/s rd, 0 op/s wr 2021-04-04T09:43:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:18.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster1 service dump 2021-04-04T09:43:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2021-04-04T09:43:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "epoch": 11, 2021-04-04T09:43:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "modified": "2021-04-04T09:40:29.716259+0000", 2021-04-04T09:43:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "services": { 2021-04-04T09:43:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "rbd-mirror": { 2021-04-04T09:43:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "daemons": { 2021-04-04T09:43:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "summary": "", 2021-04-04T09:43:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4934": { 2021-04-04T09:43:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_epoch": 11, 2021-04-04T09:43:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_stamp": "2021-04-04T09:40:29.087999+0000", 2021-04-04T09:43:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "gid": 4934, 2021-04-04T09:43:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "addr": "172.21.15.122:0/3613086818", 2021-04-04T09:43:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "metadata": { 2021-04-04T09:43:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "arch": "x86_64", 2021-04-04T09:43:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_release": "pacific", 2021-04-04T09:43:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version": "ceph version 16.2.0-22-gf4fb4141 (f4fb41419a42bee43a2069637ef17064073bf30b) pacific (stable)", 2021-04-04T09:43:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version_short": "16.2.0-22-gf4fb4141", 2021-04-04T09:43:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-04-04T09:43:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro": "ubuntu", 2021-04-04T09:43:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2021-04-04T09:43:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_version": "18.04", 2021-04-04T09:43:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "hostname": "smithi122", 2021-04-04T09:43:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "id": "mirror.0", 2021-04-04T09:43:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4934", 2021-04-04T09:43:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_description": "#144-Ubuntu SMP Fri Mar 19 14:12:35 UTC 2021", 2021-04-04T09:43:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_version": "4.15.0-140-generic", 2021-04-04T09:43:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_swap_kb": "2097148", 2021-04-04T09:43:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_total_kb": "32823744", 2021-04-04T09:43:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "os": "Linux" 2021-04-04T09:43:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "task_status": {} 2021-04-04T09:43:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4938": { 2021-04-04T09:43:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_epoch": 11, 2021-04-04T09:43:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_stamp": "2021-04-04T09:40:29.104200+0000", 2021-04-04T09:43:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "gid": 4938, 2021-04-04T09:43:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "addr": "172.21.15.122:0/75107711", 2021-04-04T09:43:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "metadata": { 2021-04-04T09:43:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "arch": "x86_64", 2021-04-04T09:43:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_release": "pacific", 2021-04-04T09:43:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version": "ceph version 16.2.0-22-gf4fb4141 (f4fb41419a42bee43a2069637ef17064073bf30b) pacific (stable)", 2021-04-04T09:43:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version_short": "16.2.0-22-gf4fb4141", 2021-04-04T09:43:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-04-04T09:43:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro": "ubuntu", 2021-04-04T09:43:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2021-04-04T09:43:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_version": "18.04", 2021-04-04T09:43:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "hostname": "smithi122", 2021-04-04T09:43:18.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "id": "mirror.1", 2021-04-04T09:43:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4938", 2021-04-04T09:43:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_description": "#144-Ubuntu SMP Fri Mar 19 14:12:35 UTC 2021", 2021-04-04T09:43:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_version": "4.15.0-140-generic", 2021-04-04T09:43:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_swap_kb": "2097148", 2021-04-04T09:43:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_total_kb": "32823744", 2021-04-04T09:43:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "os": "Linux" 2021-04-04T09:43:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "task_status": {} 2021-04-04T09:43:18.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4945": { 2021-04-04T09:43:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_epoch": 11, 2021-04-04T09:43:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_stamp": "2021-04-04T09:40:29.141998+0000", 2021-04-04T09:43:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "gid": 4945, 2021-04-04T09:43:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "addr": "172.21.15.122:0/4031853056", 2021-04-04T09:43:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "metadata": { 2021-04-04T09:43:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "arch": "x86_64", 2021-04-04T09:43:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_release": "pacific", 2021-04-04T09:43:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version": "ceph version 16.2.0-22-gf4fb4141 (f4fb41419a42bee43a2069637ef17064073bf30b) pacific (stable)", 2021-04-04T09:43:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version_short": "16.2.0-22-gf4fb4141", 2021-04-04T09:43:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-04-04T09:43:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro": "ubuntu", 2021-04-04T09:43:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2021-04-04T09:43:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_version": "18.04", 2021-04-04T09:43:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "hostname": "smithi122", 2021-04-04T09:43:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "id": "mirror.2", 2021-04-04T09:43:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4945", 2021-04-04T09:43:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_description": "#144-Ubuntu SMP Fri Mar 19 14:12:35 UTC 2021", 2021-04-04T09:43:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_version": "4.15.0-140-generic", 2021-04-04T09:43:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_swap_kb": "2097148", 2021-04-04T09:43:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_total_kb": "32823744", 2021-04-04T09:43:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "os": "Linux" 2021-04-04T09:43:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "task_status": {} 2021-04-04T09:43:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4952": { 2021-04-04T09:43:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_epoch": 11, 2021-04-04T09:43:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_stamp": "2021-04-04T09:40:29.179687+0000", 2021-04-04T09:43:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "gid": 4952, 2021-04-04T09:43:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "addr": "172.21.15.122:0/1332119204", 2021-04-04T09:43:18.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "metadata": { 2021-04-04T09:43:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "arch": "x86_64", 2021-04-04T09:43:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_release": "pacific", 2021-04-04T09:43:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version": "ceph version 16.2.0-22-gf4fb4141 (f4fb41419a42bee43a2069637ef17064073bf30b) pacific (stable)", 2021-04-04T09:43:18.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version_short": "16.2.0-22-gf4fb4141", 2021-04-04T09:43:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-04-04T09:43:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro": "ubuntu", 2021-04-04T09:43:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2021-04-04T09:43:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_version": "18.04", 2021-04-04T09:43:18.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "hostname": "smithi122", 2021-04-04T09:43:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "id": "mirror.3", 2021-04-04T09:43:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4952", 2021-04-04T09:43:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_description": "#144-Ubuntu SMP Fri Mar 19 14:12:35 UTC 2021", 2021-04-04T09:43:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_version": "4.15.0-140-generic", 2021-04-04T09:43:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_swap_kb": "2097148", 2021-04-04T09:43:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_total_kb": "32823744", 2021-04-04T09:43:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "os": "Linux" 2021-04-04T09:43:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:18.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "task_status": {} 2021-04-04T09:43:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:18.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2021-04-04T09:43:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster1 service status 2021-04-04T09:43:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2021-04-04T09:43:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "rbd-mirror": { 2021-04-04T09:43:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4934": { 2021-04-04T09:43:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status_stamp": "2021-04-04T09:43:14.096200+0000", 2021-04-04T09:43:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "last_beacon": "2021-04-04T09:43:14.096200+0000", 2021-04-04T09:43:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status": { 2021-04-04T09:43:18.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":4,\"image_error_count\":0,\"image_local_count\":10,\"image_remote_count\":9,\"image_warning_count\":0,\"instance_id\":\"4944\",\"leader\":true,\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":2,\"image_remote_count\":1,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":1,\"image_remote_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":1,\"image_error_count\":0,\"image_local_count\":1,\"image_remote_count\":1,\"image_warning_count\":0,\"instance_id\":\"4964\",\"leader\":true}}" 2021-04-04T09:43:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4938": { 2021-04-04T09:43:18.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status_stamp": "2021-04-04T09:43:14.112667+0000", 2021-04-04T09:43:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "last_beacon": "2021-04-04T09:43:14.112667+0000", 2021-04-04T09:43:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status": { 2021-04-04T09:43:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":4,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4948\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":1,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{}}}" 2021-04-04T09:43:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4945": { 2021-04-04T09:43:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status_stamp": "2021-04-04T09:43:14.149074+0000", 2021-04-04T09:43:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "last_beacon": "2021-04-04T09:43:14.149074+0000", 2021-04-04T09:43:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status": { 2021-04-04T09:43:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":1,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4957\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4975\"}}" 2021-04-04T09:43:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4952": { 2021-04-04T09:43:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status_stamp": "2021-04-04T09:43:14.186095+0000", 2021-04-04T09:43:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "last_beacon": "2021-04-04T09:43:14.186095+0000", 2021-04-04T09:43:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status": { 2021-04-04T09:43:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4961\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4971\"}}" 2021-04-04T09:43:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:18.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2021-04-04T09:43:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:18.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror images 2021-04-04T09:43:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror images 2021-04-04T09:43:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ls -l 2021-04-04T09:43:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:NAME SIZE PARENT FMT PROT LOCK 2021-04-04T09:43:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test 128 MiB 2 excl 2021-04-04T09:43:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test1 128 MiB 2 excl 2021-04-04T09:43:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2 128 MiB 2 excl 2021-04-04T09:43:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2@snap1 128 MiB 2 2021-04-04T09:43:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2@snap2 128 MiB 2 2021-04-04T09:43:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2@snap_rename_20 128 MiB 2 2021-04-04T09:43:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test4 128 MiB 2 excl 2021-04-04T09:43:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test4@snap1 128 MiB 2 yes 2021-04-04T09:43:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test4@snap2 128 MiB 2 yes 2021-04-04T09:43:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone 128 MiB mirror_parent/test_parent@snap 2 excl 2021-04-04T09:43:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone1 128 MiB mirror_parent/test_parent@snap 2 2021-04-04T09:43:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone_v1 128 MiB mirror_parent/test_parent@snap 2 excl 2021-04-04T09:43:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool 128 MiB 2 excl 2021-04-04T09:43:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool@snap1 128 MiB 2 2021-04-04T09:43:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool@snap2 128 MiB 2 2021-04-04T09:43:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_force_promote 128 MiB 2 2021-04-04T09:43:18.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:18.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:18.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror mirror pool info 2021-04-04T09:43:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror mirror pool info 2021-04-04T09:43:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace mirror pool info 2021-04-04T09:43:18.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: pool 2021-04-04T09:43:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Site Name: cluster1 2021-04-04T09:43:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Peer Sites: 2021-04-04T09:43:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:UUID: ba5e7ea1-e930-4550-b570-945e322d5ab4 2021-04-04T09:43:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Name: cluster2 2021-04-04T09:43:18.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirror UUID: f36635b8-4737-4365-816f-3d9690ce95a7 2021-04-04T09:43:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Direction: rx-tx 2021-04-04T09:43:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Client: client.mirror 2021-04-04T09:43:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror mirror pool status 2021-04-04T09:43:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror mirror pool status 2021-04-04T09:43:18.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror --namespace mirror pool status --verbose 2021-04-04T09:43:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: WARNING 2021-04-04T09:43:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: OK 2021-04-04T09:43:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: WARNING 2021-04-04T09:43:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 9 total 2021-04-04T09:43:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 9 unknown 2021-04-04T09:43:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2021-04-04T09:43:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4934: 2021-04-04T09:43:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4944 2021-04-04T09:43:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.0 2021-04-04T09:43:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T09:43:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T09:43:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: true 2021-04-04T09:43:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T09:43:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4938: 2021-04-04T09:43:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4948 2021-04-04T09:43:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.1 2021-04-04T09:43:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T09:43:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T09:43:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T09:43:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T09:43:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4945: 2021-04-04T09:43:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4957 2021-04-04T09:43:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.2 2021-04-04T09:43:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T09:43:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T09:43:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T09:43:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T09:43:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4952: 2021-04-04T09:43:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4961 2021-04-04T09:43:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.3 2021-04-04T09:43:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T09:43:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T09:43:19.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T09:43:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T09:43:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2021-04-04T09:43:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test: 2021-04-04T09:43:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 3c8112b2-6f20-4a55-833f-e3a0d1e92beb 2021-04-04T09:43:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:19.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T09:43:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2021-04-04T09:43:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T09:43:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test1: 2021-04-04T09:43:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:43:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:43:19.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:43:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T09:43:19.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2: 2021-04-04T09:43:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 80ed7469-0640-4a03-9952-a34e4d433832 2021-04-04T09:43:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":22},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":22}} 2021-04-04T09:43:19.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:43:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T09:43:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test4: 2021-04-04T09:43:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 698bca8d-5596-41e0-b6a2-71fc39245e79 2021-04-04T09:43:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:19.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T09:43:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:43:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T09:43:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:19.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:19.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone: 2021-04-04T09:43:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: a9fc372c-84b2-41cd-a172-f5e0c63e6d90 2021-04-04T09:43:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:19.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:43:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:43:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T09:43:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone1: 2021-04-04T09:43:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 14c06da4-7ebf-41e7-8aa7-0b0a1328684a 2021-04-04T09:43:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:43:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: local image is primary 2021-04-04T09:43:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:43:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:43:14 2021-04-04T09:43:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T09:43:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone_v1: 2021-04-04T09:43:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: caa231b2-f85a-456d-b241-95227b74117e 2021-04-04T09:43:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:19.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T09:43:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:43:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T09:43:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool: 2021-04-04T09:43:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 63d2ca14-0fae-4993-96ee-cffbec8f3ffc 2021-04-04T09:43:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T09:43:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:43:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T09:43:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_force_promote: 2021-04-04T09:43:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T09:43:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:43:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: local image is primary 2021-04-04T09:43:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:43:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:43:14 2021-04-04T09:43:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T09:43:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ls 2021-04-04T09:43:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test info 2021-04-04T09:43:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test info 2021-04-04T09:43:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test 2021-04-04T09:43:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2021-04-04T09:43:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T09:43:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 135d95a808e2 2021-04-04T09:43:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.135d95a808e2 2021-04-04T09:43:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:19.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:40:43 2021 2021-04-04T09:43:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:40:43 2021 2021-04-04T09:43:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:40:43 2021 2021-04-04T09:43:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 135d95a808e2 2021-04-04T09:43:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:19.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:19.314 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 3c8112b2-6f20-4a55-833f-e3a0d1e92beb 2021-04-04T09:43:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:43:19.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.316 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test journal status 2021-04-04T09:43:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test journal status 2021-04-04T09:43:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test 2021-04-04T09:43:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:19.318 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:19.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test snapshots 2021-04-04T09:43:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test snapshots 2021-04-04T09:43:19.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test 2021-04-04T09:43:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test1 info 2021-04-04T09:43:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test1 info 2021-04-04T09:43:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test1 2021-04-04T09:43:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test1': 2021-04-04T09:43:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T09:43:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10dd683f3cad 2021-04-04T09:43:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10dd683f3cad 2021-04-04T09:43:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:19.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:31:07 2021 2021-04-04T09:43:19.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:31:07 2021 2021-04-04T09:43:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:31:07 2021 2021-04-04T09:43:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10dd683f3cad 2021-04-04T09:43:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:43:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:43:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test1 journal status 2021-04-04T09:43:19.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test1 journal status 2021-04-04T09:43:19.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test1 2021-04-04T09:43:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test1 snapshots 2021-04-04T09:43:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test1 snapshots 2021-04-04T09:43:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test1 2021-04-04T09:43:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test2 info 2021-04-04T09:43:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test2 info 2021-04-04T09:43:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test2 2021-04-04T09:43:19.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test2': 2021-04-04T09:43:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:19.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 3 2021-04-04T09:43:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0b93cfe9e 2021-04-04T09:43:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0b93cfe9e 2021-04-04T09:43:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:35:24 2021 2021-04-04T09:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:35:24 2021 2021-04-04T09:43:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:35:24 2021 2021-04-04T09:43:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10e0b93cfe9e 2021-04-04T09:43:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 80ed7469-0640-4a03-9952-a34e4d433832 2021-04-04T09:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:43:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test2 journal status 2021-04-04T09:43:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test2 journal status 2021-04-04T09:43:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test2 2021-04-04T09:43:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test2 snapshots 2021-04-04T09:43:19.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test2 snapshots 2021-04-04T09:43:19.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test2 2021-04-04T09:43:19.772 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T09:43:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 44 snap1 128 MiB Sun Apr 4 09:35:25 2021 user 2021-04-04T09:43:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 45 snap2 128 MiB Sun Apr 4 09:35:25 2021 user 2021-04-04T09:43:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 52 snap_rename_20 128 MiB Sun Apr 4 09:39:55 2021 user 2021-04-04T09:43:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test4 info 2021-04-04T09:43:19.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test4 info 2021-04-04T09:43:19.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test4 2021-04-04T09:43:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test4': 2021-04-04T09:43:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:19.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2021-04-04T09:43:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e78fad30ae 2021-04-04T09:43:19.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e78fad30ae 2021-04-04T09:43:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:35:31 2021 2021-04-04T09:43:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:35:31 2021 2021-04-04T09:43:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:35:31 2021 2021-04-04T09:43:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10e78fad30ae 2021-04-04T09:43:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 698bca8d-5596-41e0-b6a2-71fc39245e79 2021-04-04T09:43:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:43:19.851 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test4 journal status 2021-04-04T09:43:19.852 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test4 journal status 2021-04-04T09:43:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test4 2021-04-04T09:43:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:19.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:19.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:19.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test4 snapshots 2021-04-04T09:43:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test4 snapshots 2021-04-04T09:43:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test4 2021-04-04T09:43:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T09:43:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 48 snap1 128 MiB yes Sun Apr 4 09:35:32 2021 user 2021-04-04T09:43:19.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 49 snap2 128 MiB yes Sun Apr 4 09:35:33 2021 user 2021-04-04T09:43:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone info 2021-04-04T09:43:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone info 2021-04-04T09:43:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test_clone 2021-04-04T09:43:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_clone': 2021-04-04T09:43:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T09:43:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ddbd07e7b5 2021-04-04T09:43:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ddbd07e7b5 2021-04-04T09:43:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:05 2021 2021-04-04T09:43:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:05 2021 2021-04-04T09:43:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:05 2021 2021-04-04T09:43:20.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: parent: mirror_parent/test_parent@snap 2021-04-04T09:43:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: overlap: 128 MiB 2021-04-04T09:43:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10ddbd07e7b5 2021-04-04T09:43:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:20.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: a9fc372c-84b2-41cd-a172-f5e0c63e6d90 2021-04-04T09:43:20.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:43:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone journal status 2021-04-04T09:43:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone journal status 2021-04-04T09:43:20.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test_clone 2021-04-04T09:43:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:20.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:20.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:20.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone snapshots 2021-04-04T09:43:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone snapshots 2021-04-04T09:43:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test_clone 2021-04-04T09:43:20.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone1 info 2021-04-04T09:43:20.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone1 info 2021-04-04T09:43:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test_clone1 2021-04-04T09:43:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_clone1': 2021-04-04T09:43:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T09:43:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12487c3dac7b 2021-04-04T09:43:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12487c3dac7b 2021-04-04T09:43:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:12 2021 2021-04-04T09:43:20.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:12 2021 2021-04-04T09:43:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:12 2021 2021-04-04T09:43:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: parent: mirror_parent/test_parent@snap 2021-04-04T09:43:20.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: overlap: 128 MiB 2021-04-04T09:43:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 12487c3dac7b 2021-04-04T09:43:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 14c06da4-7ebf-41e7-8aa7-0b0a1328684a 2021-04-04T09:43:20.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T09:43:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone1 journal status 2021-04-04T09:43:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone1 journal status 2021-04-04T09:43:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test_clone1 2021-04-04T09:43:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:20.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T09:43:20.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T09:43:20.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone1 snapshots 2021-04-04T09:43:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone1 snapshots 2021-04-04T09:43:20.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test_clone1 2021-04-04T09:43:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone_v1 info 2021-04-04T09:43:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone_v1 info 2021-04-04T09:43:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test_clone_v1 2021-04-04T09:43:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_clone_v1': 2021-04-04T09:43:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T09:43:20.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e71c96086d 2021-04-04T09:43:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e71c96086d 2021-04-04T09:43:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:20.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:14 2021 2021-04-04T09:43:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:14 2021 2021-04-04T09:43:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:14 2021 2021-04-04T09:43:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: parent: mirror_parent/test_parent@snap 2021-04-04T09:43:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: overlap: 128 MiB 2021-04-04T09:43:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10e71c96086d 2021-04-04T09:43:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: caa231b2-f85a-456d-b241-95227b74117e 2021-04-04T09:43:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:43:20.648 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:20.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone_v1 journal status 2021-04-04T09:43:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone_v1 journal status 2021-04-04T09:43:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test_clone_v1 2021-04-04T09:43:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:20.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:20.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone_v1 snapshots 2021-04-04T09:43:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone_v1 snapshots 2021-04-04T09:43:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test_clone_v1 2021-04-04T09:43:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:20.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool info 2021-04-04T09:43:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool info 2021-04-04T09:43:20.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test_data_pool 2021-04-04T09:43:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_data_pool': 2021-04-04T09:43:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:20.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2021-04-04T09:43:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ddcfb530aa 2021-04-04T09:43:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: data_pool: mirror_parent 2021-04-04T09:43:20.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.3.10ddcfb530aa 2021-04-04T09:43:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling, data-pool 2021-04-04T09:43:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:39 2021 2021-04-04T09:43:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:39 2021 2021-04-04T09:43:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:39 2021 2021-04-04T09:43:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10ddcfb530aa 2021-04-04T09:43:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:20.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 63d2ca14-0fae-4993-96ee-cffbec8f3ffc 2021-04-04T09:43:20.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:43:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool journal status 2021-04-04T09:43:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool journal status 2021-04-04T09:43:20.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test_data_pool 2021-04-04T09:43:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool snapshots 2021-04-04T09:43:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool snapshots 2021-04-04T09:43:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test_data_pool 2021-04-04T09:43:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T09:43:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 14 snap1 128 MiB Sun Apr 4 09:34:44 2021 user 2021-04-04T09:43:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 15 snap2 128 MiB Sun Apr 4 09:34:46 2021 user 2021-04-04T09:43:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_force_promote info 2021-04-04T09:43:21.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_force_promote info 2021-04-04T09:43:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test_force_promote 2021-04-04T09:43:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_force_promote': 2021-04-04T09:43:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T09:43:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ea33c092e 2021-04-04T09:43:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ea33c092e 2021-04-04T09:43:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:33:43 2021 2021-04-04T09:43:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:33:43 2021 2021-04-04T09:43:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:33:43 2021 2021-04-04T09:43:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10ea33c092e 2021-04-04T09:43:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T09:43:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T09:43:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_force_promote journal status 2021-04-04T09:43:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_force_promote journal status 2021-04-04T09:43:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test_force_promote 2021-04-04T09:43:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=101], [object_number=0, tag_tid=3, entry_tid=100], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected] 2021-04-04T09:43:21.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_force_promote snapshots 2021-04-04T09:43:21.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_force_promote snapshots 2021-04-04T09:43:21.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test_force_promote 2021-04-04T09:43:21.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror rbd_mirroring omap vals 2021-04-04T09:43:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror rbd_mirroring omap vals 2021-04-04T09:43:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror --namespace listomapvals rbd_mirroring 2021-04-04T09:43:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_14c06da4-7ebf-41e7-8aa7-0b0a1328684a 2021-04-04T09:43:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 34 38 37 63 33 64 61 63 37 62 |....12487c3dac7b| 2021-04-04T09:43:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_3c8112b2-6f20-4a55-833f-e3a0d1e92beb 2021-04-04T09:43:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 33 35 64 39 35 61 38 30 38 65 32 |....135d95a808e2| 2021-04-04T09:43:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T09:43:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (15 bytes) : 2021-04-04T09:43:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0b 00 00 00 31 30 65 61 33 33 63 30 39 32 65 |....10ea33c092e| 2021-04-04T09:43:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000000f 2021-04-04T09:43:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_63d2ca14-0fae-4993-96ee-cffbec8f3ffc 2021-04-04T09:43:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 64 64 63 66 62 35 33 30 61 61 |....10ddcfb530aa| 2021-04-04T09:43:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_698bca8d-5596-41e0-b6a2-71fc39245e79 2021-04-04T09:43:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 37 38 66 61 64 33 30 61 65 |....10e78fad30ae| 2021-04-04T09:43:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_80ed7469-0640-4a03-9952-a34e4d433832 2021-04-04T09:43:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 30 62 39 33 63 66 65 39 65 |....10e0b93cfe9e| 2021-04-04T09:43:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_a9fc372c-84b2-41cd-a172-f5e0c63e6d90 2021-04-04T09:43:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 64 64 62 64 30 37 65 37 62 35 |....10ddbd07e7b5| 2021-04-04T09:43:21.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_caa231b2-f85a-456d-b241-95227b74117e 2021-04-04T09:43:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 37 31 63 39 36 30 38 36 64 |....10e71c96086d| 2021-04-04T09:43:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:43:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 64 64 36 38 33 66 33 63 61 64 |....10dd683f3cad| 2021-04-04T09:43:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10dd683f3cad 2021-04-04T09:43:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 31 61 62 65 35 |..*...$...e1abe5| 2021-04-04T09:43:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 63 35 2d 61 37 37 33 2d 34 39 65 34 2d 39 34 30 |c5-a773-49e4-940| 2021-04-04T09:43:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 39 2d 62 32 62 66 37 39 63 66 64 65 33 31 01 00 |9-b2bf79cfde31..| 2021-04-04T09:43:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ddbd07e7b5 2021-04-04T09:43:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:21.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 61 39 66 63 33 37 |..*...$...a9fc37| 2021-04-04T09:43:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 32 63 2d 38 34 62 32 2d 34 31 63 64 2d 61 31 37 |2c-84b2-41cd-a17| 2021-04-04T09:43:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 2d 66 35 65 30 63 36 33 65 36 64 39 30 01 00 |2-f5e0c63e6d90..| 2021-04-04T09:43:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ddcfb530aa 2021-04-04T09:43:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 33 64 32 63 61 |..*...$...63d2ca| 2021-04-04T09:43:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 31 34 2d 30 66 61 65 2d 34 39 39 33 2d 39 36 65 |14-0fae-4993-96e| 2021-04-04T09:43:21.337 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 65 2d 63 66 66 62 65 63 38 66 33 66 66 63 01 00 |e-cffbec8f3ffc..| 2021-04-04T09:43:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e0b93cfe9e 2021-04-04T09:43:21.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 30 65 64 37 34 |..*...$...80ed74| 2021-04-04T09:43:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 36 39 2d 30 36 34 30 2d 34 61 30 33 2d 39 39 35 |69-0640-4a03-995| 2021-04-04T09:43:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 2d 61 33 34 65 34 64 34 33 33 38 33 32 01 00 |2-a34e4d433832..| 2021-04-04T09:43:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:21.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e71c96086d 2021-04-04T09:43:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 61 61 32 33 31 |..*...$...caa231| 2021-04-04T09:43:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 62 32 2d 66 38 35 61 2d 34 35 36 64 2d 62 32 34 |b2-f85a-456d-b24| 2021-04-04T09:43:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 31 2d 39 35 32 32 37 62 37 34 31 31 37 65 01 00 |1-95227b74117e..| 2021-04-04T09:43:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e78fad30ae 2021-04-04T09:43:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 39 38 62 63 61 |..*...$...698bca| 2021-04-04T09:43:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 64 2d 35 35 39 36 2d 34 31 65 30 2d 62 36 61 |8d-5596-41e0-b6a| 2021-04-04T09:43:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 2d 37 31 66 63 33 39 32 34 35 65 37 39 01 00 |2-71fc39245e79..| 2021-04-04T09:43:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ea33c092e 2021-04-04T09:43:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 33 39 65 37 36 |..*...$...439e76| 2021-04-04T09:43:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 66 37 2d 65 39 63 62 2d 34 66 39 32 2d 38 31 30 |f7-e9cb-4f92-810| 2021-04-04T09:43:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 64 2d 63 66 37 65 39 35 33 30 33 64 37 39 01 00 |d-cf7e95303d79..| 2021-04-04T09:43:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12487c3dac7b 2021-04-04T09:43:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 34 63 30 36 64 |..*...$...14c06d| 2021-04-04T09:43:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 61 34 2d 37 65 62 66 2d 34 31 65 37 2d 38 61 61 |a4-7ebf-41e7-8aa| 2021-04-04T09:43:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 37 2d 30 62 30 61 31 33 32 38 36 38 34 61 01 00 |7-0b0a1328684a..| 2021-04-04T09:43:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_135d95a808e2 2021-04-04T09:43:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 33 63 38 31 31 32 |..*...$...3c8112| 2021-04-04T09:43:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 62 32 2d 36 66 32 30 2d 34 61 35 35 2d 38 33 33 |b2-6f20-4a55-833| 2021-04-04T09:43:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 66 2d 65 33 61 30 64 31 65 39 32 62 65 62 01 00 |f-e3a0d1e92beb..| 2021-04-04T09:43:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2021-04-04T09:43:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2021-04-04T09:43:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 00 00 00 |....| 2021-04-04T09:43:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2021-04-04T09:43:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_peer_ba5e7ea1-e930-4550-b570-945e322d5ab4 2021-04-04T09:43:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (132 bytes) : 2021-04-04T09:43:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 62 61 35 65 37 65 |..~...$...ba5e7e| 2021-04-04T09:43:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 61 31 2d 65 39 33 30 2d 34 35 35 30 2d 62 35 37 |a1-e930-4550-b57| 2021-04-04T09:43:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 30 2d 39 34 35 65 33 32 32 64 35 61 62 34 08 00 |0-945e322d5ab4..| 2021-04-04T09:43:21.349 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2021-04-04T09:43:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2021-04-04T09:43:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 ff ff ff 02 24 00 00 00 66 33 36 36 33 35 62 38 |....$...f36635b8| 2021-04-04T09:43:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2d 34 37 33 37 2d 34 33 36 35 2d 38 31 36 66 2d |-4737-4365-816f-| 2021-04-04T09:43:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 33 64 39 36 39 30 63 65 39 35 61 37 8b 89 69 60 |3d9690ce95a7..i`| 2021-04-04T09:43:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 ec 6e 99 2d |.n.-| 2021-04-04T09:43:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000084 2021-04-04T09:43:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2021-04-04T09:43:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2021-04-04T09:43:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 34 65 63 62 63 64 32 37 2d 64 33 35 31 2d 34 34 |4ecbcd27-d351-44| 2021-04-04T09:43:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 35 30 2d 39 33 32 36 2d 32 37 61 39 36 66 33 35 |50-9326-27a96f35| 2021-04-04T09:43:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 34 39 33 |2493| 2021-04-04T09:43:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2021-04-04T09:43:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_09a6fbfb-f68f-4e71-98d3-0b0c50ff9d74 2021-04-04T09:43:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T09:43:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T09:43:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 f5 d2 c9 0d 00 | replay..i`.....| 2021-04-04T09:43:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_14c06da4-7ebf-41e7-8aa7-0b0a1328684a 2021-04-04T09:43:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (92 bytes) : 2021-04-04T09:43:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T09:43:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T09:43:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2021-04-04T09:43:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2021-04-04T09:43:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 61 72 79 37 8a 69 60 d3 dd bc 0c 00 |ary7.i`.....| 2021-04-04T09:43:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000005c 2021-04-04T09:43:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_19dd5345-25d8-4554-8034-6c31e0df778e 2021-04-04T09:43:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T09:43:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T09:43:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 d5 4a 41 0e 00 | replay..i`.JA..| 2021-04-04T09:43:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_3c8112b2-6f20-4a55-833f-e3a0d1e92beb 2021-04-04T09:43:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T09:43:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 5d 13 00 00 00 00 00 00 01 |..,....]........| 2021-04-04T09:43:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2d 21 8d f0 10 00 |..........-!....| 2021-04-04T09:43:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T09:43:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T09:43:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T09:43:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T09:43:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T09:43:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T09:43:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 31 7d 2c 22 70 72 69 6d 61 72 |_tid":1},"primar| 2021-04-04T09:43:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T09:43:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T09:43:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T09:43:21.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 31 7d 7d 23 8a 69 60 b6 2e |g_tid":1}}#.i`..| 2021-04-04T09:43:21.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 f2 0d 00 |...| 2021-04-04T09:43:21.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T09:43:21.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_409477f3-cbd2-4eff-9e17-c87e509f4e40 2021-04-04T09:43:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T09:43:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T09:43:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 ee 85 fb 0d 00 | replay..i`.....| 2021-04-04T09:43:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T09:43:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (92 bytes) : 2021-04-04T09:43:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T09:43:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T09:43:21.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2021-04-04T09:43:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2021-04-04T09:43:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 61 72 79 37 8a 69 60 9c d4 ac 0b 00 |ary7.i`.....| 2021-04-04T09:43:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000005c 2021-04-04T09:43:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_63d2ca14-0fae-4993-96ee-cffbec8f3ffc 2021-04-04T09:43:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (295 bytes) : 2021-04-04T09:43:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T09:43:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T09:43:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 ef 00 00 00 04 e1 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T09:43:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T09:43:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T09:43:21.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T09:43:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 31 30 31 2c 22 6f 62 6a |ry_tid":101,"obj| 2021-04-04T09:43:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 65 63 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 |ect_number":1,"t| 2021-04-04T09:43:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 61 67 5f 74 69 64 22 3a 35 7d 2c 22 70 72 69 6d |ag_tid":5},"prim| 2021-04-04T09:43:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 72 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 |ary_position":{"| 2021-04-04T09:43:21.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 65 6e 74 72 79 5f 74 69 64 22 3a 31 30 31 2c 22 |entry_tid":101,"| 2021-04-04T09:43:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 6f 62 6a 65 63 74 5f 6e 75 6d 62 65 72 22 3a 31 |object_number":1| 2021-04-04T09:43:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 2c 22 74 61 67 5f 74 69 64 22 3a 35 7d 7d 23 8a |,"tag_tid":5}}#.| 2021-04-04T09:43:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 69 60 06 4a 2d 0b 00 |i`.J-..| 2021-04-04T09:43:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000127 2021-04-04T09:43:21.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_690ce498-199d-4023-9c64-ff4d7599e591 2021-04-04T09:43:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T09:43:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T09:43:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 31 a7 41 0e 00 | replay..i`1.A..| 2021-04-04T09:43:21.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_698bca8d-5596-41e0-b6a2-71fc39245e79 2021-04-04T09:43:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T09:43:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T09:43:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T09:43:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T09:43:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T09:43:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T09:43:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T09:43:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T09:43:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T09:43:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 31 7d 2c 22 70 72 69 6d 61 72 |_tid":1},"primar| 2021-04-04T09:43:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T09:43:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T09:43:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T09:43:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 31 7d 7d 23 8a 69 60 cf 60 |g_tid":1}}#.i`.`| 2021-04-04T09:43:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 2e 0b 00 |...| 2021-04-04T09:43:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T09:43:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_80ed7469-0640-4a03-9952-a34e4d433832 2021-04-04T09:43:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (293 bytes) : 2021-04-04T09:43:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T09:43:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T09:43:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 ed 00 00 00 04 df 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:21.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T09:43:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T09:43:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T09:43:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T09:43:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 31 2c 22 6f 62 6a 65 63 |ry_tid":1,"objec| 2021-04-04T09:43:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 61 67 |t_number":1,"tag| 2021-04-04T09:43:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 32 32 7d 2c 22 70 72 69 6d 61 |_tid":22},"prima| 2021-04-04T09:43:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 72 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 |ry_position":{"e| 2021-04-04T09:43:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 6e 74 72 79 5f 74 69 64 22 3a 31 2c 22 6f 62 6a |ntry_tid":1,"obj| 2021-04-04T09:43:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 65 63 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 |ect_number":1,"t| 2021-04-04T09:43:21.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 61 67 5f 74 69 64 22 3a 32 32 7d 7d 23 8a 69 60 |ag_tid":22}}#.i`| 2021-04-04T09:43:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 9a 8c 12 0a 00 |.....| 2021-04-04T09:43:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000125 2021-04-04T09:43:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_87727dd3-4985-41eb-bd8e-3591e277f847 2021-04-04T09:43:21.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T09:43:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T09:43:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:21.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 43 72 5e 0e 00 | replay..i`Cr^..| 2021-04-04T09:43:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_a744e3fd-757f-4a7c-b80e-63f68a69a413 2021-04-04T09:43:21.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T09:43:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T09:43:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:21.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 27 b6 7c 0e 00 | replay..i`'.|..| 2021-04-04T09:43:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_a9fc372c-84b2-41cd-a172-f5e0c63e6d90 2021-04-04T09:43:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T09:43:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T09:43:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T09:43:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T09:43:21.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T09:43:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T09:43:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T09:43:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T09:43:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T09:43:21.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 32 7d 2c 22 70 72 69 6d 61 72 |_tid":2},"primar| 2021-04-04T09:43:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T09:43:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T09:43:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T09:43:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 32 7d 7d 23 8a 69 60 09 35 |g_tid":2}}#.i`.5| 2021-04-04T09:43:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 2f 0b 00 |/..| 2021-04-04T09:43:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T09:43:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_caa231b2-f85a-456d-b241-95227b74117e 2021-04-04T09:43:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T09:43:21.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T09:43:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T09:43:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T09:43:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T09:43:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T09:43:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T09:43:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T09:43:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T09:43:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 31 7d 2c 22 70 72 69 6d 61 72 |_tid":1},"primar| 2021-04-04T09:43:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T09:43:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T09:43:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T09:43:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 31 7d 7d 23 8a 69 60 6c 84 |g_tid":1}}#.i`l.| 2021-04-04T09:43:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 14 0a 00 |...| 2021-04-04T09:43:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T09:43:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T09:43:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T09:43:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T09:43:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 14 85 92 0e 00 | replay..i`.....| 2021-04-04T09:43:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:43:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T09:43:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T09:43:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T09:43:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T09:43:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T09:43:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T09:43:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T09:43:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T09:43:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T09:43:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 32 7d 2c 22 70 72 69 6d 61 72 |_tid":2},"primar| 2021-04-04T09:43:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T09:43:21.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T09:43:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T09:43:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 32 7d 7d 23 8a 69 60 44 26 |g_tid":2}}#.i`D&| 2021-04-04T09:43:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 16 0a 00 |...| 2021-04-04T09:43:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T09:43:21.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T09:43:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T09:43:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T09:43:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 92 89 69 60 20 4e 2e 0c 00 | replay..i` N...| 2021-04-04T09:43:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:21.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror ns1 images 2021-04-04T09:43:21.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror ns1 images 2021-04-04T09:43:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 ls -l 2021-04-04T09:43:21.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:NAME SIZE PARENT FMT PROT LOCK 2021-04-04T09:43:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool 128 MiB 2 excl 2021-04-04T09:43:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool@.rbd-mirror.4ecbcd27-d351-4450-9326-27a96f352493.2fc90865-f9af-418c-899c-52a5812d98c1 128 MiB 2 2021-04-04T09:43:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirrorns1 mirror pool info 2021-04-04T09:43:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirrorns1 mirror pool info 2021-04-04T09:43:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool info 2021-04-04T09:43:21.435 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: pool 2021-04-04T09:43:21.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirrorns1 mirror pool status 2021-04-04T09:43:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirrorns1 mirror pool status 2021-04-04T09:43:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool status --verbose 2021-04-04T09:43:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: OK 2021-04-04T09:43:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: OK 2021-04-04T09:43:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: OK 2021-04-04T09:43:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 1 total 2021-04-04T09:43:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 1 replaying 2021-04-04T09:43:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2021-04-04T09:43:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4934: 2021-04-04T09:43:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4944 2021-04-04T09:43:21.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.0 2021-04-04T09:43:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T09:43:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T09:43:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: true 2021-04-04T09:43:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T09:43:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4938: 2021-04-04T09:43:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4948 2021-04-04T09:43:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.1 2021-04-04T09:43:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T09:43:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T09:43:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T09:43:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T09:43:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4945: 2021-04-04T09:43:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4957 2021-04-04T09:43:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.2 2021-04-04T09:43:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T09:43:21.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T09:43:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T09:43:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T09:43:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4952: 2021-04-04T09:43:21.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4961 2021-04-04T09:43:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.3 2021-04-04T09:43:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T09:43:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T09:43:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T09:43:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T09:43:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2021-04-04T09:43:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool: 2021-04-04T09:43:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 41466285-b62b-4d8b-b1e4-3b04bb3f224c 2021-04-04T09:43:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":104,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":2}} 2021-04-04T09:43:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T09:43:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T09:43:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 ls 2021-04-04T09:43:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool info 2021-04-04T09:43:21.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool info 2021-04-04T09:43:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 info test_data_pool 2021-04-04T09:43:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_data_pool': 2021-04-04T09:43:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2021-04-04T09:43:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1354875c93da 2021-04-04T09:43:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: data_pool: mirror_parent 2021-04-04T09:43:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.3.1354875c93da 2021-04-04T09:43:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling, data-pool 2021-04-04T09:43:21.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:41:01 2021 2021-04-04T09:43:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:41:01 2021 2021-04-04T09:43:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:41:01 2021 2021-04-04T09:43:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 1354875c93da 2021-04-04T09:43:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 41466285-b62b-4d8b-b1e4-3b04bb3f224c 2021-04-04T09:43:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:43:21.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool journal status 2021-04-04T09:43:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:21.642 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool journal status 2021-04-04T09:43:21.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 journal status --image test_data_pool 2021-04-04T09:43:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool snapshots 2021-04-04T09:43:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool snapshots 2021-04-04T09:43:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 snap ls --all test_data_pool 2021-04-04T09:43:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T09:43:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 16 .rbd-mirror.4ecbcd27-d351-4450-9326-27a96f352493.2fc90865-f9af-418c-899c-52a5812d98c1 128 MiB Sun Apr 4 09:41:01 2021 user 2021-04-04T09:43:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror ns1 rbd_mirroring omap vals 2021-04-04T09:43:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror ns1 rbd_mirroring omap vals 2021-04-04T09:43:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirroring 2021-04-04T09:43:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_41466285-b62b-4d8b-b1e4-3b04bb3f224c 2021-04-04T09:43:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 33 35 34 38 37 35 63 39 33 64 61 |....1354875c93da| 2021-04-04T09:43:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_1354875c93da 2021-04-04T09:43:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 31 34 36 36 32 |..*...$...414662| 2021-04-04T09:43:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 35 2d 62 36 32 62 2d 34 64 38 62 2d 62 31 65 |85-b62b-4d8b-b1e| 2021-04-04T09:43:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 34 2d 33 62 30 34 62 62 33 66 32 32 34 63 01 00 |4-3b04bb3f224c..| 2021-04-04T09:43:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2021-04-04T09:43:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2021-04-04T09:43:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 00 00 00 |....| 2021-04-04T09:43:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2021-04-04T09:43:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2021-04-04T09:43:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2021-04-04T09:43:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 35 35 30 63 35 61 65 62 2d 66 31 64 39 2d 34 38 |550c5aeb-f1d9-48| 2021-04-04T09:43:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 32 31 2d 61 39 31 35 2d 34 34 30 37 33 30 31 63 |21-a915-4407301c| 2021-04-04T09:43:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 65 38 31 34 |e814| 2021-04-04T09:43:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2021-04-04T09:43:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_41466285-b62b-4d8b-b1e4-3b04bb3f224c 2021-04-04T09:43:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (252 bytes) : 2021-04-04T09:43:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T09:43:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T09:43:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 c4 00 00 00 04 b6 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 31 30 34 2c 22 65 6e |primary":104,"en| 2021-04-04T09:43:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 74 72 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 |tries_per_second| 2021-04-04T09:43:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 22 3a 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 |":0.0,"non_prima| 2021-04-04T09:43:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 72 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 7d 2c |ry_position":{},| 2021-04-04T09:43:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 70 72 69 6d 61 72 79 5f 70 6f 73 69 74 69 6f |"primary_positio| 2021-04-04T09:43:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 6e 22 3a 7b 22 65 6e 74 72 79 5f 74 69 64 22 3a |n":{"entry_tid":| 2021-04-04T09:43:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 30 31 2c 22 6f 62 6a 65 63 74 5f 6e 75 6d 62 |101,"object_numb| 2021-04-04T09:43:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 65 72 22 3a 31 2c 22 74 61 67 5f 74 69 64 22 3a |er":1,"tag_tid":| 2021-04-04T09:43:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 32 7d 7d 23 8a 69 60 51 7a fc 0c 00 |2}}#.i`Qz...| 2021-04-04T09:43:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000fc 2021-04-04T09:43:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_909bceaa-c260-415a-a60e-eee31d4f3e89 2021-04-04T09:43:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T09:43:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T09:43:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 ab 89 69 60 28 2d 6f 0c 00 | replay..i`(-o..| 2021-04-04T09:43:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror ns2 images 2021-04-04T09:43:21.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror ns2 images 2021-04-04T09:43:21.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 ls -l 2021-04-04T09:43:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirrorns2 mirror pool info 2021-04-04T09:43:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirrorns2 mirror pool info 2021-04-04T09:43:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool info 2021-04-04T09:43:21.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: image 2021-04-04T09:43:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirrorns2 mirror pool status 2021-04-04T09:43:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirrorns2 mirror pool status 2021-04-04T09:43:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool status --verbose 2021-04-04T09:43:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: OK 2021-04-04T09:43:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: OK 2021-04-04T09:43:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: OK 2021-04-04T09:43:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 0 total 2021-04-04T09:43:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2021-04-04T09:43:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4934: 2021-04-04T09:43:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4944 2021-04-04T09:43:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.0 2021-04-04T09:43:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T09:43:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T09:43:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: true 2021-04-04T09:43:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T09:43:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4938: 2021-04-04T09:43:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4948 2021-04-04T09:43:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.1 2021-04-04T09:43:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T09:43:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T09:43:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T09:43:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T09:43:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4945: 2021-04-04T09:43:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4957 2021-04-04T09:43:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.2 2021-04-04T09:43:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T09:43:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T09:43:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T09:43:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T09:43:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4952: 2021-04-04T09:43:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4961 2021-04-04T09:43:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.3 2021-04-04T09:43:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T09:43:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T09:43:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T09:43:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T09:43:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.980 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2021-04-04T09:43:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 ls 2021-04-04T09:43:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror ns2 rbd_mirroring omap vals 2021-04-04T09:43:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror ns2 rbd_mirroring omap vals 2021-04-04T09:43:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirroring 2021-04-04T09:43:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2021-04-04T09:43:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2021-04-04T09:43:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 00 00 00 |....| 2021-04-04T09:43:22.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2021-04-04T09:43:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2021-04-04T09:43:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2021-04-04T09:43:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 30 32 33 35 62 38 37 31 2d 36 38 63 39 2d 34 65 |0235b871-68c9-4e| 2021-04-04T09:43:22.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 62 61 2d 39 30 62 62 2d 62 34 65 38 36 64 30 32 |ba-90bb-b4e86d02| 2021-04-04T09:43:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 38 35 31 |2851| 2021-04-04T09:43:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2021-04-04T09:43:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_af2694d6-07b0-4e37-95e1-92663262de9a 2021-04-04T09:43:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T09:43:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T09:43:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:22.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 ab 89 69 60 19 33 7b 0c 00 | replay..i`.3{..| 2021-04-04T09:43:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:22.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent images 2021-04-04T09:43:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parent images 2021-04-04T09:43:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ls -l 2021-04-04T09:43:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:NAME SIZE PARENT FMT PROT LOCK 2021-04-04T09:43:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_parent 128 MiB 2 excl 2021-04-04T09:43:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_parent@snap 128 MiB 2 yes 2021-04-04T09:43:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent mirror pool info 2021-04-04T09:43:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parent mirror pool info 2021-04-04T09:43:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace mirror pool info 2021-04-04T09:43:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: image 2021-04-04T09:43:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Site Name: cluster1 2021-04-04T09:43:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Peer Sites: 2021-04-04T09:43:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:UUID: 062cbc91-4713-4ded-b744-6fda24ecf93a 2021-04-04T09:43:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Name: cluster2 2021-04-04T09:43:22.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirror UUID: de536629-eb1e-4d40-bad3-7bb7b2b463fd 2021-04-04T09:43:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Direction: rx-tx 2021-04-04T09:43:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Client: client.mirror 2021-04-04T09:43:22.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent mirror pool status 2021-04-04T09:43:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parent mirror pool status 2021-04-04T09:43:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror_parent --namespace mirror pool status --verbose 2021-04-04T09:43:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: WARNING 2021-04-04T09:43:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: OK 2021-04-04T09:43:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: WARNING 2021-04-04T09:43:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 1 total 2021-04-04T09:43:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 1 unknown 2021-04-04T09:43:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2021-04-04T09:43:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4934: 2021-04-04T09:43:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4964 2021-04-04T09:43:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.0 2021-04-04T09:43:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T09:43:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T09:43:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: true 2021-04-04T09:43:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T09:43:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4938: 2021-04-04T09:43:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 2021-04-04T09:43:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.1 2021-04-04T09:43:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T09:43:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T09:43:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T09:43:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T09:43:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4945: 2021-04-04T09:43:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4975 2021-04-04T09:43:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.2 2021-04-04T09:43:22.230 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T09:43:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T09:43:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T09:43:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T09:43:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4952: 2021-04-04T09:43:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4971 2021-04-04T09:43:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.3 2021-04-04T09:43:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T09:43:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T09:43:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T09:43:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T09:43:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2021-04-04T09:43:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_parent: 2021-04-04T09:43:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: ed3379f2-3a26-4e23-8dd8-e2567a8361ce 2021-04-04T09:43:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":8},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":8}} 2021-04-04T09:43:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T09:43:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T09:43:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ls 2021-04-04T09:43:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_parent info 2021-04-04T09:43:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_parent info 2021-04-04T09:43:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace info test_parent 2021-04-04T09:43:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_parent': 2021-04-04T09:43:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2021-04-04T09:43:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ed3aaf7c64 2021-04-04T09:43:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ed3aaf7c64 2021-04-04T09:43:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:33:58 2021 2021-04-04T09:43:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:33:58 2021 2021-04-04T09:43:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:33:58 2021 2021-04-04T09:43:22.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10ed3aaf7c64 2021-04-04T09:43:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:22.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: ed3379f2-3a26-4e23-8dd8-e2567a8361ce 2021-04-04T09:43:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:43:22.383 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_parent journal status 2021-04-04T09:43:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_parent journal status 2021-04-04T09:43:22.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace journal status --image test_parent 2021-04-04T09:43:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:22.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:22.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=de536629-eb1e-4d40-bad3-7bb7b2b463fd, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_parent snapshots 2021-04-04T09:43:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_parent snapshots 2021-04-04T09:43:22.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace snap ls --all test_parent 2021-04-04T09:43:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T09:43:22.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 4 snap 128 MiB yes Sun Apr 4 09:33:59 2021 user 2021-04-04T09:43:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent rbd_mirroring omap vals 2021-04-04T09:43:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parent rbd_mirroring omap vals 2021-04-04T09:43:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace listomapvals rbd_mirroring 2021-04-04T09:43:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_ed3379f2-3a26-4e23-8dd8-e2567a8361ce 2021-04-04T09:43:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 64 33 61 61 66 37 63 36 34 |....10ed3aaf7c64| 2021-04-04T09:43:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ed3aaf7c64 2021-04-04T09:43:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 64 33 33 37 39 |..*...$...ed3379| 2021-04-04T09:43:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 66 32 2d 33 61 32 36 2d 34 65 32 33 2d 38 64 64 |f2-3a26-4e23-8dd| 2021-04-04T09:43:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 38 2d 65 32 35 36 37 61 38 33 36 31 63 65 01 00 |8-e2567a8361ce..| 2021-04-04T09:43:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2021-04-04T09:43:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2021-04-04T09:43:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 00 00 00 |....| 2021-04-04T09:43:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2021-04-04T09:43:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_peer_062cbc91-4713-4ded-b744-6fda24ecf93a 2021-04-04T09:43:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (132 bytes) : 2021-04-04T09:43:22.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 30 36 32 63 62 63 |..~...$...062cbc| 2021-04-04T09:43:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 39 31 2d 34 37 31 33 2d 34 64 65 64 2d 62 37 34 |91-4713-4ded-b74| 2021-04-04T09:43:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 34 2d 36 66 64 61 32 34 65 63 66 39 33 61 08 00 |4-6fda24ecf93a..| 2021-04-04T09:43:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2021-04-04T09:43:22.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2021-04-04T09:43:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 ff ff ff 02 24 00 00 00 64 65 35 33 36 36 32 39 |....$...de536629| 2021-04-04T09:43:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2d 65 62 31 65 2d 34 64 34 30 2d 62 61 64 33 2d |-eb1e-4d40-bad3-| 2021-04-04T09:43:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 37 62 62 37 62 32 62 34 36 33 66 64 6e 89 69 60 |7bb7b2b463fdn.i`| 2021-04-04T09:43:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 ab cd 1a 02 |....| 2021-04-04T09:43:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000084 2021-04-04T09:43:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2021-04-04T09:43:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2021-04-04T09:43:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 31 63 64 32 31 66 65 33 2d 62 65 33 62 2d 34 61 |1cd21fe3-be3b-4a| 2021-04-04T09:43:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 35 38 2d 61 63 32 64 2d 61 32 35 64 65 33 31 38 |58-ac2d-a25de318| 2021-04-04T09:43:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 64 65 62 63 |debc| 2021-04-04T09:43:22.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2021-04-04T09:43:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_ed3379f2-3a26-4e23-8dd8-e2567a8361ce 2021-04-04T09:43:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T09:43:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 64 13 00 00 00 00 00 00 01 |..,....d........| 2021-04-04T09:43:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 d4 ba e4 e5 10 00 |................| 2021-04-04T09:43:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T09:43:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T09:43:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T09:43:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T09:43:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 31 2c 22 6f 62 6a 65 63 |ry_tid":1,"objec| 2021-04-04T09:43:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 61 67 |t_number":1,"tag| 2021-04-04T09:43:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 38 7d 2c 22 70 72 69 6d 61 72 |_tid":8},"primar| 2021-04-04T09:43:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T09:43:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 31 2c 22 6f 62 6a 65 |try_tid":1,"obje| 2021-04-04T09:43:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 61 |ct_number":1,"ta| 2021-04-04T09:43:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 38 7d 7d 23 8a 69 60 d1 45 |g_tid":8}}#.i`.E| 2021-04-04T09:43:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 7b 10 00 |{..| 2021-04-04T09:43:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T09:43:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parent ns1 images 2021-04-04T09:43:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent ns1 images 2021-04-04T09:43:22.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls -l 2021-04-04T09:43:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns1' does not exist. 2021-04-04T09:43:22.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: listing images failed: (2) No such file or directory 2021-04-04T09:43:22.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parentns1 mirror pool info 2021-04-04T09:43:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parentns1 mirror pool info 2021-04-04T09:43:22.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool info 2021-04-04T09:43:22.641 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns1' does not exist. 2021-04-04T09:43:22.643 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.644 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parentns1 mirror pool status 2021-04-04T09:43:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parentns1 mirror pool status 2021-04-04T09:43:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool status --verbose 2021-04-04T09:43:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns1' does not exist. 2021-04-04T09:43:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls 2021-04-04T09:43:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parent ns1 rbd_mirroring omap vals 2021-04-04T09:43:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent ns1 rbd_mirroring omap vals 2021-04-04T09:43:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2021-04-04T09:43:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2021-04-04T09:43:22.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:22.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent ns2 images 2021-04-04T09:43:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parent ns2 images 2021-04-04T09:43:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls -l 2021-04-04T09:43:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns2' does not exist. 2021-04-04T09:43:22.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: listing images failed: (2) No such file or directory 2021-04-04T09:43:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parentns2 mirror pool info 2021-04-04T09:43:22.803 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parentns2 mirror pool info 2021-04-04T09:43:22.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool info 2021-04-04T09:43:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns2' does not exist. 2021-04-04T09:43:22.844 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parentns2 mirror pool status 2021-04-04T09:43:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parentns2 mirror pool status 2021-04-04T09:43:22.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool status --verbose 2021-04-04T09:43:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns2' does not exist. 2021-04-04T09:43:22.889 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:22.890 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls 2021-04-04T09:43:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent ns2 rbd_mirroring omap vals 2021-04-04T09:43:22.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parent ns2 rbd_mirroring omap vals 2021-04-04T09:43:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2021-04-04T09:43:22.968 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2021-04-04T09:43:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 status 2021-04-04T09:43:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 status 2021-04-04T09:43:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster2 -s 2021-04-04T09:43:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: cluster: 2021-04-04T09:43:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 0fed4950-22e0-476b-9583-b4115f5b795c 2021-04-04T09:43:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: HEALTH_OK 2021-04-04T09:43:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: services: 2021-04-04T09:43:23.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mon: 1 daemons, quorum a (age 13m) 2021-04-04T09:43:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mgr: x(active, since 13m) 2021-04-04T09:43:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: osd: 3 osds: 3 up (since 13m), 3 in (since 13m) 2021-04-04T09:43:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:23.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: data: 2021-04-04T09:43:23.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: pools: 4 pools, 137 pgs 2021-04-04T09:43:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: objects: 502 objects, 1.0 GiB 2021-04-04T09:43:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: usage: 1.4 GiB used, 267 GiB / 268 GiB avail 2021-04-04T09:43:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: pgs: 137 active+clean 2021-04-04T09:43:23.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: io: 2021-04-04T09:43:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client: 21 KiB/s rd, 341 B/s wr, 25 op/s rd, 0 op/s wr 2021-04-04T09:43:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:23.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster2 service dump 2021-04-04T09:43:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2021-04-04T09:43:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "epoch": 8, 2021-04-04T09:43:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "modified": "2021-04-04T09:41:04.241619+0000", 2021-04-04T09:43:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "services": {} 2021-04-04T09:43:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2021-04-04T09:43:23.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster2 service status 2021-04-04T09:43:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{} 2021-04-04T09:43:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror images 2021-04-04T09:43:24.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror images 2021-04-04T09:43:24.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ls -l 2021-04-04T09:43:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:NAME SIZE PARENT FMT PROT LOCK 2021-04-04T09:43:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test 128 MiB 2 2021-04-04T09:43:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test1 128 MiB 2 2021-04-04T09:43:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2 128 MiB 2 2021-04-04T09:43:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2@snap1 128 MiB 2 2021-04-04T09:43:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2@snap2 128 MiB 2 2021-04-04T09:43:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2@snap_rename_20 128 MiB 2 2021-04-04T09:43:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test4 128 MiB 2 2021-04-04T09:43:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test4@snap1 128 MiB 2 yes 2021-04-04T09:43:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test4@snap2 128 MiB 2 yes 2021-04-04T09:43:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone 128 MiB mirror_parent/test_parent@snap 2 2021-04-04T09:43:24.373 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone1 128 MiB mirror_parent/test_parent@snap 2 2021-04-04T09:43:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone_v1 128 MiB mirror_parent/test_parent@snap 2 2021-04-04T09:43:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool 128 MiB 2 2021-04-04T09:43:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool@snap1 128 MiB 2 2021-04-04T09:43:24.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool@snap2 128 MiB 2 2021-04-04T09:43:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_force_promote 128 MiB 2 2021-04-04T09:43:24.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror mirror pool info 2021-04-04T09:43:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror mirror pool info 2021-04-04T09:43:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace mirror pool info 2021-04-04T09:43:24.406 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: pool 2021-04-04T09:43:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Site Name: cluster2 2021-04-04T09:43:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Peer Sites: 2021-04-04T09:43:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:UUID: ba5c4a7e-49df-4b8d-99f2-b5377df755f6 2021-04-04T09:43:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Name: cluster1 2021-04-04T09:43:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirror UUID: 4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Direction: rx-tx 2021-04-04T09:43:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Client: client.mirror 2021-04-04T09:43:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror mirror pool status 2021-04-04T09:43:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror mirror pool status 2021-04-04T09:43:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror --namespace mirror pool status --verbose 2021-04-04T09:43:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: UNKNOWN 2021-04-04T09:43:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: UNKNOWN 2021-04-04T09:43:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: OK 2021-04-04T09:43:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 9 total 2021-04-04T09:43:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 7 replaying 2021-04-04T09:43:24.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2 stopped 2021-04-04T09:43:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2021-04-04T09:43:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: none 2021-04-04T09:43:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2021-04-04T09:43:24.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test: 2021-04-04T09:43:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 3c8112b2-6f20-4a55-833f-e3a0d1e92beb 2021-04-04T09:43:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:24.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T09:43:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T09:43:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test1: 2021-04-04T09:43:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:43:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T09:43:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:43:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2: 2021-04-04T09:43:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 80ed7469-0640-4a03-9952-a34e4d433832 2021-04-04T09:43:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:24.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:24.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:24.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T09:43:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":22},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":22}} 2021-04-04T09:43:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test4: 2021-04-04T09:43:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 698bca8d-5596-41e0-b6a2-71fc39245e79 2021-04-04T09:43:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T09:43:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T09:43:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone: 2021-04-04T09:43:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: a9fc372c-84b2-41cd-a172-f5e0c63e6d90 2021-04-04T09:43:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T09:43:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T09:43:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone1: 2021-04-04T09:43:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 14c06da4-7ebf-41e7-8aa7-0b0a1328684a 2021-04-04T09:43:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T09:43:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:43:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: local image is primary 2021-04-04T09:43:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:43:24 2021-04-04T09:43:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone_v1: 2021-04-04T09:43:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: caa231b2-f85a-456d-b241-95227b74117e 2021-04-04T09:43:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T09:43:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T09:43:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool: 2021-04-04T09:43:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 63d2ca14-0fae-4993-96ee-cffbec8f3ffc 2021-04-04T09:43:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T09:43:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T09:43:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_force_promote: 2021-04-04T09:43:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T09:43:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T09:43:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T09:43:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: local image is primary 2021-04-04T09:43:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:43:24 2021-04-04T09:43:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ls 2021-04-04T09:43:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test info 2021-04-04T09:43:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test info 2021-04-04T09:43:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test 2021-04-04T09:43:24.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2021-04-04T09:43:24.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T09:43:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1099ce7c2fd0 2021-04-04T09:43:24.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.1099ce7c2fd0 2021-04-04T09:43:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:30:50 2021 2021-04-04T09:43:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:40:44 2021 2021-04-04T09:43:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:32:37 2021 2021-04-04T09:43:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 1099ce7c2fd0 2021-04-04T09:43:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 3c8112b2-6f20-4a55-833f-e3a0d1e92beb 2021-04-04T09:43:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T09:43:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test journal status 2021-04-04T09:43:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test journal status 2021-04-04T09:43:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test 2021-04-04T09:43:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T09:43:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T09:43:24.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test snapshots 2021-04-04T09:43:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test snapshots 2021-04-04T09:43:24.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test 2021-04-04T09:43:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test1 info 2021-04-04T09:43:24.724 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test1 info 2021-04-04T09:43:24.725 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test1 2021-04-04T09:43:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test1': 2021-04-04T09:43:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T09:43:24.774 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e1f6d670b3 2021-04-04T09:43:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e1f6d670b3 2021-04-04T09:43:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:24.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:31:05 2021 2021-04-04T09:43:24.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:31:05 2021 2021-04-04T09:43:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:31:05 2021 2021-04-04T09:43:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10e1f6d670b3 2021-04-04T09:43:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:43:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T09:43:24.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test1 journal status 2021-04-04T09:43:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test1 journal status 2021-04-04T09:43:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test1 2021-04-04T09:43:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:43:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:43:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test1 snapshots 2021-04-04T09:43:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test1 snapshots 2021-04-04T09:43:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test1 2021-04-04T09:43:24.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test2 info 2021-04-04T09:43:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test2 info 2021-04-04T09:43:24.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test2 2021-04-04T09:43:24.952 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test2': 2021-04-04T09:43:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 3 2021-04-04T09:43:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12c761e38e1d 2021-04-04T09:43:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12c761e38e1d 2021-04-04T09:43:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:50 2021 2021-04-04T09:43:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:50 2021 2021-04-04T09:43:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:50 2021 2021-04-04T09:43:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 12c761e38e1d 2021-04-04T09:43:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 80ed7469-0640-4a03-9952-a34e4d433832 2021-04-04T09:43:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T09:43:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test2 journal status 2021-04-04T09:43:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test2 journal status 2021-04-04T09:43:24.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test2 2021-04-04T09:43:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=1, tag_tid=22, entry_tid=1], [object_number=0, tag_tid=22, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2]]], state=connected] 2021-04-04T09:43:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=22, entry_tid=1], [object_number=0, tag_tid=22, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2]]], state=connected] 2021-04-04T09:43:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test2 snapshots 2021-04-04T09:43:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test2 snapshots 2021-04-04T09:43:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test2 2021-04-04T09:43:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T09:43:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 20 snap1 128 MiB Sun Apr 4 09:34:51 2021 user 2021-04-04T09:43:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 21 snap2 128 MiB Sun Apr 4 09:34:52 2021 user 2021-04-04T09:43:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 44 snap_rename_20 128 MiB Sun Apr 4 09:39:55 2021 user 2021-04-04T09:43:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test4 info 2021-04-04T09:43:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test4 info 2021-04-04T09:43:25.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test4 2021-04-04T09:43:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test4': 2021-04-04T09:43:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2021-04-04T09:43:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12e57718fb70 2021-04-04T09:43:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12e57718fb70 2021-04-04T09:43:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:35:01 2021 2021-04-04T09:43:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:35:01 2021 2021-04-04T09:43:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:35:01 2021 2021-04-04T09:43:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 12e57718fb70 2021-04-04T09:43:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 698bca8d-5596-41e0-b6a2-71fc39245e79 2021-04-04T09:43:25.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T09:43:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test4 journal status 2021-04-04T09:43:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test4 journal status 2021-04-04T09:43:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test4 2021-04-04T09:43:25.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T09:43:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T09:43:25.245 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test4 snapshots 2021-04-04T09:43:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:25.246 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test4 snapshots 2021-04-04T09:43:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test4 2021-04-04T09:43:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T09:43:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 28 snap1 128 MiB yes Sun Apr 4 09:35:02 2021 user 2021-04-04T09:43:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 29 snap2 128 MiB yes Sun Apr 4 09:35:03 2021 user 2021-04-04T09:43:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone info 2021-04-04T09:43:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone info 2021-04-04T09:43:25.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test_clone 2021-04-04T09:43:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_clone': 2021-04-04T09:43:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T09:43:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1226aee074ef 2021-04-04T09:43:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.1226aee074ef 2021-04-04T09:43:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:25.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:33:56 2021 2021-04-04T09:43:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:33:56 2021 2021-04-04T09:43:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:33:56 2021 2021-04-04T09:43:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: parent: mirror_parent/test_parent@snap 2021-04-04T09:43:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: overlap: 128 MiB 2021-04-04T09:43:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 1226aee074ef 2021-04-04T09:43:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: a9fc372c-84b2-41cd-a172-f5e0c63e6d90 2021-04-04T09:43:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T09:43:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone journal status 2021-04-04T09:43:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone journal status 2021-04-04T09:43:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test_clone 2021-04-04T09:43:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:43:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:43:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:25.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone snapshots 2021-04-04T09:43:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone snapshots 2021-04-04T09:43:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test_clone 2021-04-04T09:43:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone1 info 2021-04-04T09:43:25.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone1 info 2021-04-04T09:43:25.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test_clone1 2021-04-04T09:43:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_clone1': 2021-04-04T09:43:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T09:43:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 113b92c6dbf7 2021-04-04T09:43:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.113b92c6dbf7 2021-04-04T09:43:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:25.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:25.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:25.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:25.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:14 2021 2021-04-04T09:43:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:14 2021 2021-04-04T09:43:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:14 2021 2021-04-04T09:43:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: parent: mirror_parent/test_parent@snap 2021-04-04T09:43:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: overlap: 128 MiB 2021-04-04T09:43:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 113b92c6dbf7 2021-04-04T09:43:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 14c06da4-7ebf-41e7-8aa7-0b0a1328684a 2021-04-04T09:43:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T09:43:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone1 journal status 2021-04-04T09:43:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone1 journal status 2021-04-04T09:43:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test_clone1 2021-04-04T09:43:25.716 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:25.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:25.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone1 snapshots 2021-04-04T09:43:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone1 snapshots 2021-04-04T09:43:25.721 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test_clone1 2021-04-04T09:43:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone_v1 info 2021-04-04T09:43:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone_v1 info 2021-04-04T09:43:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test_clone_v1 2021-04-04T09:43:25.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_clone_v1': 2021-04-04T09:43:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T09:43:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 125dca854283 2021-04-04T09:43:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.125dca854283 2021-04-04T09:43:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:12 2021 2021-04-04T09:43:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:12 2021 2021-04-04T09:43:25.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:12 2021 2021-04-04T09:43:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: parent: mirror_parent/test_parent@snap 2021-04-04T09:43:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: overlap: 128 MiB 2021-04-04T09:43:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 125dca854283 2021-04-04T09:43:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: caa231b2-f85a-456d-b241-95227b74117e 2021-04-04T09:43:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T09:43:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone_v1 journal status 2021-04-04T09:43:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone_v1 journal status 2021-04-04T09:43:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test_clone_v1 2021-04-04T09:43:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T09:43:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T09:43:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone_v1 snapshots 2021-04-04T09:43:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone_v1 snapshots 2021-04-04T09:43:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test_clone_v1 2021-04-04T09:43:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool info 2021-04-04T09:43:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:26.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool info 2021-04-04T09:43:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test_data_pool 2021-04-04T09:43:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_data_pool': 2021-04-04T09:43:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2021-04-04T09:43:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1284d6a187ce 2021-04-04T09:43:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: data_pool: mirror_parent 2021-04-04T09:43:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.3.1284d6a187ce 2021-04-04T09:43:26.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling, data-pool 2021-04-04T09:43:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:37 2021 2021-04-04T09:43:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:37 2021 2021-04-04T09:43:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:37 2021 2021-04-04T09:43:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 1284d6a187ce 2021-04-04T09:43:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:26.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 63d2ca14-0fae-4993-96ee-cffbec8f3ffc 2021-04-04T09:43:26.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T09:43:26.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool journal status 2021-04-04T09:43:26.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool journal status 2021-04-04T09:43:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test_data_pool 2021-04-04T09:43:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T09:43:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T09:43:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool snapshots 2021-04-04T09:43:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool snapshots 2021-04-04T09:43:26.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test_data_pool 2021-04-04T09:43:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T09:43:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 13 snap1 128 MiB Sun Apr 4 09:34:44 2021 user 2021-04-04T09:43:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 14 snap2 128 MiB Sun Apr 4 09:34:45 2021 user 2021-04-04T09:43:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_force_promote info 2021-04-04T09:43:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_force_promote info 2021-04-04T09:43:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test_force_promote 2021-04-04T09:43:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_force_promote': 2021-04-04T09:43:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:26.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T09:43:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 11fbaec917d8 2021-04-04T09:43:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.11fbaec917d8 2021-04-04T09:43:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:33:41 2021 2021-04-04T09:43:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:33:41 2021 2021-04-04T09:43:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:33:41 2021 2021-04-04T09:43:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 11fbaec917d8 2021-04-04T09:43:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T09:43:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T09:43:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_force_promote journal status 2021-04-04T09:43:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_force_promote journal status 2021-04-04T09:43:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test_force_promote 2021-04-04T09:43:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:26.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=101], [object_number=0, tag_tid=3, entry_tid=100], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected] 2021-04-04T09:43:26.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T09:43:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_force_promote snapshots 2021-04-04T09:43:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_force_promote snapshots 2021-04-04T09:43:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test_force_promote 2021-04-04T09:43:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror rbd_mirroring omap vals 2021-04-04T09:43:26.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror rbd_mirroring omap vals 2021-04-04T09:43:26.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror --namespace listomapvals rbd_mirroring 2021-04-04T09:43:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_14c06da4-7ebf-41e7-8aa7-0b0a1328684a 2021-04-04T09:43:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 31 33 62 39 32 63 36 64 62 66 37 |....113b92c6dbf7| 2021-04-04T09:43:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_3c8112b2-6f20-4a55-833f-e3a0d1e92beb 2021-04-04T09:43:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 39 39 63 65 37 63 32 66 64 30 |....1099ce7c2fd0| 2021-04-04T09:43:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T09:43:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 31 66 62 61 65 63 39 31 37 64 38 |....11fbaec917d8| 2021-04-04T09:43:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_63d2ca14-0fae-4993-96ee-cffbec8f3ffc 2021-04-04T09:43:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 38 34 64 36 61 31 38 37 63 65 |....1284d6a187ce| 2021-04-04T09:43:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_698bca8d-5596-41e0-b6a2-71fc39245e79 2021-04-04T09:43:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 65 35 37 37 31 38 66 62 37 30 |....12e57718fb70| 2021-04-04T09:43:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_80ed7469-0640-4a03-9952-a34e4d433832 2021-04-04T09:43:26.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 63 37 36 31 65 33 38 65 31 64 |....12c761e38e1d| 2021-04-04T09:43:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_a9fc372c-84b2-41cd-a172-f5e0c63e6d90 2021-04-04T09:43:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 32 36 61 65 65 30 37 34 65 66 |....1226aee074ef| 2021-04-04T09:43:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_caa231b2-f85a-456d-b241-95227b74117e 2021-04-04T09:43:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:26.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 35 64 63 61 38 35 34 32 38 33 |....125dca854283| 2021-04-04T09:43:26.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:26.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T09:43:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 31 66 36 64 36 37 30 62 33 |....10e1f6d670b3| 2021-04-04T09:43:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_1099ce7c2fd0 2021-04-04T09:43:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 33 63 38 31 31 32 |..*...$...3c8112| 2021-04-04T09:43:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 62 32 2d 36 66 32 30 2d 34 61 35 35 2d 38 33 33 |b2-6f20-4a55-833| 2021-04-04T09:43:26.482 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 66 2d 65 33 61 30 64 31 65 39 32 62 65 62 01 00 |f-e3a0d1e92beb..| 2021-04-04T09:43:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e1f6d670b3 2021-04-04T09:43:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 31 61 62 65 35 |..*...$...e1abe5| 2021-04-04T09:43:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 63 35 2d 61 37 37 33 2d 34 39 65 34 2d 39 34 30 |c5-a773-49e4-940| 2021-04-04T09:43:26.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 39 2d 62 32 62 66 37 39 63 66 64 65 33 31 01 00 |9-b2bf79cfde31..| 2021-04-04T09:43:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:26.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_113b92c6dbf7 2021-04-04T09:43:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 34 63 30 36 64 |..*...$...14c06d| 2021-04-04T09:43:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 61 34 2d 37 65 62 66 2d 34 31 65 37 2d 38 61 61 |a4-7ebf-41e7-8aa| 2021-04-04T09:43:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 37 2d 30 62 30 61 31 33 32 38 36 38 34 61 01 00 |7-0b0a1328684a..| 2021-04-04T09:43:26.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_11fbaec917d8 2021-04-04T09:43:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 33 39 65 37 36 |..*...$...439e76| 2021-04-04T09:43:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 66 37 2d 65 39 63 62 2d 34 66 39 32 2d 38 31 30 |f7-e9cb-4f92-810| 2021-04-04T09:43:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 64 2d 63 66 37 65 39 35 33 30 33 64 37 39 01 00 |d-cf7e95303d79..| 2021-04-04T09:43:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_1226aee074ef 2021-04-04T09:43:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:26.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 61 39 66 63 33 37 |..*...$...a9fc37| 2021-04-04T09:43:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 32 63 2d 38 34 62 32 2d 34 31 63 64 2d 61 31 37 |2c-84b2-41cd-a17| 2021-04-04T09:43:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 2d 66 35 65 30 63 36 33 65 36 64 39 30 01 00 |2-f5e0c63e6d90..| 2021-04-04T09:43:26.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_125dca854283 2021-04-04T09:43:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 61 61 32 33 31 |..*...$...caa231| 2021-04-04T09:43:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 62 32 2d 66 38 35 61 2d 34 35 36 64 2d 62 32 34 |b2-f85a-456d-b24| 2021-04-04T09:43:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 31 2d 39 35 32 32 37 62 37 34 31 31 37 65 01 00 |1-95227b74117e..| 2021-04-04T09:43:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:26.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_1284d6a187ce 2021-04-04T09:43:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 33 64 32 63 61 |..*...$...63d2ca| 2021-04-04T09:43:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 31 34 2d 30 66 61 65 2d 34 39 39 33 2d 39 36 65 |14-0fae-4993-96e| 2021-04-04T09:43:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 65 2d 63 66 66 62 65 63 38 66 33 66 66 63 01 00 |e-cffbec8f3ffc..| 2021-04-04T09:43:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12c761e38e1d 2021-04-04T09:43:26.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 30 65 64 37 34 |..*...$...80ed74| 2021-04-04T09:43:26.498 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 36 39 2d 30 36 34 30 2d 34 61 30 33 2d 39 39 35 |69-0640-4a03-995| 2021-04-04T09:43:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 2d 61 33 34 65 34 64 34 33 33 38 33 32 01 00 |2-a34e4d433832..| 2021-04-04T09:43:26.499 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12e57718fb70 2021-04-04T09:43:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 39 38 62 63 61 |..*...$...698bca| 2021-04-04T09:43:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 64 2d 35 35 39 36 2d 34 31 65 30 2d 62 36 61 |8d-5596-41e0-b6a| 2021-04-04T09:43:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 2d 37 31 66 63 33 39 32 34 35 65 37 39 01 00 |2-71fc39245e79..| 2021-04-04T09:43:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2021-04-04T09:43:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2021-04-04T09:43:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 00 00 00 |....| 2021-04-04T09:43:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2021-04-04T09:43:26.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_peer_ba5c4a7e-49df-4b8d-99f2-b5377df755f6 2021-04-04T09:43:26.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (132 bytes) : 2021-04-04T09:43:26.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 62 61 35 63 34 61 |..~...$...ba5c4a| 2021-04-04T09:43:26.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 37 65 2d 34 39 64 66 2d 34 62 38 64 2d 39 39 66 |7e-49df-4b8d-99f| 2021-04-04T09:43:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 2d 62 35 33 37 37 64 66 37 35 35 66 36 08 00 |2-b5377df755f6..| 2021-04-04T09:43:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 63 6c 75 73 74 65 72 31 0d 00 00 00 63 6c |..cluster1....cl| 2021-04-04T09:43:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2021-04-04T09:43:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 ff ff ff 02 24 00 00 00 34 65 63 62 63 64 32 37 |....$...4ecbcd27| 2021-04-04T09:43:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2d 64 33 35 31 2d 34 34 35 30 2d 39 33 32 36 2d |-d351-4450-9326-| 2021-04-04T09:43:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 32 37 61 39 36 66 33 35 32 34 39 33 23 8a 69 60 |27a96f352493#.i`| 2021-04-04T09:43:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 2d 51 73 10 |-Qs.| 2021-04-04T09:43:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000084 2021-04-04T09:43:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2021-04-04T09:43:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2021-04-04T09:43:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 66 33 36 36 33 35 62 38 2d 34 37 33 37 2d 34 33 |f36635b8-4737-43| 2021-04-04T09:43:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 36 35 2d 38 31 36 66 2d 33 64 39 36 39 30 63 65 |65-816f-3d9690ce| 2021-04-04T09:43:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 39 35 61 37 |95a7| 2021-04-04T09:43:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2021-04-04T09:43:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_09a6fbfb-f68f-4e71-98d3-0b0c50ff9d74_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T09:43:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 d4 e6 7c 0d 00 | replay..i`..|..| 2021-04-04T09:43:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_14c06da4-7ebf-41e7-8aa7-0b0a1328684a_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (92 bytes) : 2021-04-04T09:43:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T09:43:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2021-04-04T09:43:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2021-04-04T09:43:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 61 72 79 3c 8a 69 60 18 af ad 0c 00 |ary<.i`.....| 2021-04-04T09:43:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000005c 2021-04-04T09:43:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_19dd5345-25d8-4554-8034-6c31e0df778e_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T09:43:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 0d 32 d0 0d 00 | replay..i`.2...| 2021-04-04T09:43:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:26.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_3c8112b2-6f20-4a55-833f-e3a0d1e92beb_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T09:43:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 f1 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 bc 02 0c 40 10 00 |.............@..| 2021-04-04T09:43:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T09:43:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T09:43:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T09:43:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T09:43:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T09:43:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T09:43:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 31 7d 2c 22 70 72 69 6d 61 72 |_tid":1},"primar| 2021-04-04T09:43:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T09:43:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T09:43:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T09:43:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 31 7d 7d 23 8a 69 60 b1 53 |g_tid":1}}#.i`.S| 2021-04-04T09:43:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 20 0e 00 | ..| 2021-04-04T09:43:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T09:43:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_409477f3-cbd2-4eff-9e17-c87e509f4e40_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T09:43:26.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 90 37 b1 0d 00 | replay..i`.7...| 2021-04-04T09:43:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_439e76f7-e9cb-4f92-810d-cf7e95303d79_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (92 bytes) : 2021-04-04T09:43:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T09:43:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2021-04-04T09:43:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2021-04-04T09:43:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 61 72 79 3c 8a 69 60 ad 2e aa 0b 00 |ary<.i`.....| 2021-04-04T09:43:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000005c 2021-04-04T09:43:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_63d2ca14-0fae-4993-96ee-cffbec8f3ffc_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (295 bytes) : 2021-04-04T09:43:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T09:43:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 ef 00 00 00 04 e1 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T09:43:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T09:43:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T09:43:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T09:43:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 31 30 31 2c 22 6f 62 6a |ry_tid":101,"obj| 2021-04-04T09:43:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 65 63 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 |ect_number":1,"t| 2021-04-04T09:43:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 61 67 5f 74 69 64 22 3a 35 7d 2c 22 70 72 69 6d |ag_tid":5},"prim| 2021-04-04T09:43:26.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 72 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 |ary_position":{"| 2021-04-04T09:43:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 65 6e 74 72 79 5f 74 69 64 22 3a 31 30 31 2c 22 |entry_tid":101,"| 2021-04-04T09:43:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 6f 62 6a 65 63 74 5f 6e 75 6d 62 65 72 22 3a 31 |object_number":1| 2021-04-04T09:43:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 2c 22 74 61 67 5f 74 69 64 22 3a 35 7d 7d 23 8a |,"tag_tid":5}}#.| 2021-04-04T09:43:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 69 60 55 7f c5 0b 00 |i`U....| 2021-04-04T09:43:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000127 2021-04-04T09:43:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_690ce498-199d-4023-9c64-ff4d7599e591_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T09:43:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 bc 4a f5 0d 00 | replay..i`.J...| 2021-04-04T09:43:26.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:26.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_698bca8d-5596-41e0-b6a2-71fc39245e79_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T09:43:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T09:43:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:26.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T09:43:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T09:43:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T09:43:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T09:43:26.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T09:43:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T09:43:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 31 7d 2c 22 70 72 69 6d 61 72 |_tid":1},"primar| 2021-04-04T09:43:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T09:43:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T09:43:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T09:43:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 31 7d 7d 23 8a 69 60 13 0e |g_tid":1}}#.i`..| 2021-04-04T09:43:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 c6 0b 00 |...| 2021-04-04T09:43:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T09:43:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_80ed7469-0640-4a03-9952-a34e4d433832_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (293 bytes) : 2021-04-04T09:43:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T09:43:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 ed 00 00 00 04 df 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T09:43:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T09:43:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T09:43:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T09:43:26.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 31 2c 22 6f 62 6a 65 63 |ry_tid":1,"objec| 2021-04-04T09:43:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 61 67 |t_number":1,"tag| 2021-04-04T09:43:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 32 32 7d 2c 22 70 72 69 6d 61 |_tid":22},"prima| 2021-04-04T09:43:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 72 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 |ry_position":{"e| 2021-04-04T09:43:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 6e 74 72 79 5f 74 69 64 22 3a 31 2c 22 6f 62 6a |ntry_tid":1,"obj| 2021-04-04T09:43:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 65 63 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 |ect_number":1,"t| 2021-04-04T09:43:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 61 67 5f 74 69 64 22 3a 32 32 7d 7d 23 8a 69 60 |ag_tid":22}}#.i`| 2021-04-04T09:43:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 ec 67 62 0a 00 |.gb..| 2021-04-04T09:43:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000125 2021-04-04T09:43:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_87727dd3-4985-41eb-bd8e-3591e277f847_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T09:43:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 76 8d e2 0d 00 | replay..i`v....| 2021-04-04T09:43:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_a744e3fd-757f-4a7c-b80e-63f68a69a413_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T09:43:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 3e f4 1e 0e 00 | replay..i`>....| 2021-04-04T09:43:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_a9fc372c-84b2-41cd-a172-f5e0c63e6d90_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T09:43:26.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T09:43:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:26.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:26.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:26.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T09:43:26.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T09:43:26.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T09:43:26.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T09:43:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T09:43:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T09:43:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 32 7d 2c 22 70 72 69 6d 61 72 |_tid":2},"primar| 2021-04-04T09:43:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T09:43:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T09:43:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T09:43:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 32 7d 7d 23 8a 69 60 f4 7d |g_tid":2}}#.i`.}| 2021-04-04T09:43:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 c6 0b 00 |...| 2021-04-04T09:43:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T09:43:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_caa231b2-f85a-456d-b241-95227b74117e_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T09:43:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T09:43:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:26.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T09:43:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T09:43:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T09:43:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T09:43:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T09:43:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T09:43:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 31 7d 2c 22 70 72 69 6d 61 72 |_tid":1},"primar| 2021-04-04T09:43:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T09:43:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T09:43:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T09:43:26.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 31 7d 7d 23 8a 69 60 b9 70 |g_tid":1}}#.i`.p| 2021-04-04T09:43:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 63 0a 00 |c..| 2021-04-04T09:43:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T09:43:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_d9317e9f-8ea9-4380-a618-30a1fe0a728a_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T09:43:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 e2 e7 34 0e 00 | replay..i`..4..| 2021-04-04T09:43:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_e1abe5c5-a773-49e4-9409-b2bf79cfde31_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T09:43:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T09:43:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T09:43:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T09:43:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T09:43:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T09:43:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T09:43:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T09:43:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 32 7d 2c 22 70 72 69 6d 61 72 |_tid":2},"primar| 2021-04-04T09:43:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T09:43:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T09:43:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T09:43:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 32 7d 7d 23 8a 69 60 af 85 |g_tid":2}}#.i`..| 2021-04-04T09:43:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 64 0a 00 |d..| 2021-04-04T09:43:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T09:43:26.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_ead75b84-b901-49af-ba5f-d9a8188a67ef_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T09:43:26.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 92 89 69 60 be 09 2b 0c 00 | replay..i`..+..| 2021-04-04T09:43:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:26.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror ns1 images 2021-04-04T09:43:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror ns1 images 2021-04-04T09:43:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 ls -l 2021-04-04T09:43:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:NAME SIZE PARENT FMT PROT LOCK 2021-04-04T09:43:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool 128 MiB 2 2021-04-04T09:43:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirrorns1 mirror pool info 2021-04-04T09:43:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirrorns1 mirror pool info 2021-04-04T09:43:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool info 2021-04-04T09:43:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: pool 2021-04-04T09:43:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirrorns1 mirror pool status 2021-04-04T09:43:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirrorns1 mirror pool status 2021-04-04T09:43:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool status --verbose 2021-04-04T09:43:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: WARNING 2021-04-04T09:43:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: UNKNOWN 2021-04-04T09:43:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: WARNING 2021-04-04T09:43:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 1 total 2021-04-04T09:43:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 1 unknown 2021-04-04T09:43:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2021-04-04T09:43:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: none 2021-04-04T09:43:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2021-04-04T09:43:26.636 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool: 2021-04-04T09:43:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 41466285-b62b-4d8b-b1e4-3b04bb3f224c 2021-04-04T09:43:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T09:43:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":104,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":2}} 2021-04-04T09:43:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:26.639 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 ls 2021-04-04T09:43:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool info 2021-04-04T09:43:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 info test_data_pool 2021-04-04T09:43:26.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool info 2021-04-04T09:43:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_data_pool': 2021-04-04T09:43:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T09:43:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1438178462ae 2021-04-04T09:43:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: data_pool: mirror_parent 2021-04-04T09:43:26.742 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.3.1438178462ae 2021-04-04T09:43:26.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:26.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling, data-pool 2021-04-04T09:43:26.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:26.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:40:59 2021 2021-04-04T09:43:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:40:59 2021 2021-04-04T09:43:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:40:59 2021 2021-04-04T09:43:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 1438178462ae 2021-04-04T09:43:26.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 41466285-b62b-4d8b-b1e4-3b04bb3f224c 2021-04-04T09:43:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T09:43:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool journal status 2021-04-04T09:43:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool journal status 2021-04-04T09:43:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 journal status --image test_data_pool 2021-04-04T09:43:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T09:43:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T09:43:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool snapshots 2021-04-04T09:43:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool snapshots 2021-04-04T09:43:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 snap ls --all test_data_pool 2021-04-04T09:43:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror ns1 rbd_mirroring omap vals 2021-04-04T09:43:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror ns1 rbd_mirroring omap vals 2021-04-04T09:43:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirroring 2021-04-04T09:43:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_41466285-b62b-4d8b-b1e4-3b04bb3f224c 2021-04-04T09:43:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 34 33 38 31 37 38 34 36 32 61 65 |....1438178462ae| 2021-04-04T09:43:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_1438178462ae 2021-04-04T09:43:26.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 31 34 36 36 32 |..*...$...414662| 2021-04-04T09:43:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 35 2d 62 36 32 62 2d 34 64 38 62 2d 62 31 65 |85-b62b-4d8b-b1e| 2021-04-04T09:43:26.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 34 2d 33 62 30 34 62 62 33 66 32 32 34 63 01 00 |4-3b04bb3f224c..| 2021-04-04T09:43:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2021-04-04T09:43:26.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2021-04-04T09:43:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 00 00 00 |....| 2021-04-04T09:43:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2021-04-04T09:43:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2021-04-04T09:43:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2021-04-04T09:43:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 37 39 36 34 35 34 65 61 2d 33 31 35 62 2d 34 62 |796454ea-315b-4b| 2021-04-04T09:43:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 64 65 2d 38 62 66 33 2d 31 32 36 65 33 34 31 64 |de-8bf3-126e341d| 2021-04-04T09:43:26.917 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 65 32 62 36 |e2b6| 2021-04-04T09:43:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2021-04-04T09:43:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_41466285-b62b-4d8b-b1e4-3b04bb3f224c_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.919 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (252 bytes) : 2021-04-04T09:43:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T09:43:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 c4 00 00 00 04 b6 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 31 30 34 2c 22 65 6e |primary":104,"en| 2021-04-04T09:43:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 74 72 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 |tries_per_second| 2021-04-04T09:43:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 22 3a 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 |":0.0,"non_prima| 2021-04-04T09:43:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 72 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 7d 2c |ry_position":{},| 2021-04-04T09:43:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 70 72 69 6d 61 72 79 5f 70 6f 73 69 74 69 6f |"primary_positio| 2021-04-04T09:43:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 6e 22 3a 7b 22 65 6e 74 72 79 5f 74 69 64 22 3a |n":{"entry_tid":| 2021-04-04T09:43:26.924 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 30 31 2c 22 6f 62 6a 65 63 74 5f 6e 75 6d 62 |101,"object_numb| 2021-04-04T09:43:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 65 72 22 3a 31 2c 22 74 61 67 5f 74 69 64 22 3a |er":1,"tag_tid":| 2021-04-04T09:43:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 32 7d 7d 23 8a 69 60 b1 09 0c 0d 00 |2}}#.i`.....| 2021-04-04T09:43:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000fc 2021-04-04T09:43:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_909bceaa-c260-415a-a60e-eee31d4f3e89_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:26.927 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T09:43:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 ab 89 69 60 f0 32 68 0c 00 | replay..i`.2h..| 2021-04-04T09:43:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror ns2 images 2021-04-04T09:43:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror ns2 images 2021-04-04T09:43:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 ls -l 2021-04-04T09:43:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:NAME SIZE PARENT FMT PROT LOCK 2021-04-04T09:43:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test 128 MiB 2 2021-04-04T09:43:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirrorns2 mirror pool info 2021-04-04T09:43:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirrorns2 mirror pool info 2021-04-04T09:43:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool info 2021-04-04T09:43:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: image 2021-04-04T09:43:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirrorns2 mirror pool status 2021-04-04T09:43:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirrorns2 mirror pool status 2021-04-04T09:43:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool status --verbose 2021-04-04T09:43:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: UNKNOWN 2021-04-04T09:43:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: UNKNOWN 2021-04-04T09:43:27.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: OK 2021-04-04T09:43:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 0 total 2021-04-04T09:43:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2021-04-04T09:43:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: none 2021-04-04T09:43:27.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2021-04-04T09:43:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 ls 2021-04-04T09:43:27.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test info 2021-04-04T09:43:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test info 2021-04-04T09:43:27.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 info test 2021-04-04T09:43:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2021-04-04T09:43:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T09:43:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1409c8b81b0c 2021-04-04T09:43:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.1409c8b81b0c 2021-04-04T09:43:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:27.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2021-04-04T09:43:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:40:47 2021 2021-04-04T09:43:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:40:47 2021 2021-04-04T09:43:27.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:40:47 2021 2021-04-04T09:43:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test journal status 2021-04-04T09:43:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test journal status 2021-04-04T09:43:27.387 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 journal status --image test 2021-04-04T09:43:27.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image test 2021-04-04T09:43:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test snapshots 2021-04-04T09:43:27.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test snapshots 2021-04-04T09:43:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 snap ls --all test 2021-04-04T09:43:27.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror ns2 rbd_mirroring omap vals 2021-04-04T09:43:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror ns2 rbd_mirroring omap vals 2021-04-04T09:43:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirroring 2021-04-04T09:43:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2021-04-04T09:43:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2021-04-04T09:43:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 00 00 00 |....| 2021-04-04T09:43:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2021-04-04T09:43:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2021-04-04T09:43:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2021-04-04T09:43:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 31 34 35 32 63 36 65 36 2d 62 31 31 62 2d 34 39 |1452c6e6-b11b-49| 2021-04-04T09:43:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 62 2d 61 33 61 64 2d 36 61 39 32 66 61 65 35 |8b-a3ad-6a92fae5| 2021-04-04T09:43:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 66 38 30 62 |f80b| 2021-04-04T09:43:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2021-04-04T09:43:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_af2694d6-07b0-4e37-95e1-92663262de9a_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T09:43:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T09:43:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T09:43:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T09:43:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T09:43:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 ab 89 69 60 d2 3e 71 0c 00 | replay..i`.>q..| 2021-04-04T09:43:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T09:43:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent images 2021-04-04T09:43:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parent images 2021-04-04T09:43:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ls -l 2021-04-04T09:43:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:NAME SIZE PARENT FMT PROT LOCK 2021-04-04T09:43:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_parent 128 MiB 2 2021-04-04T09:43:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_parent@snap 128 MiB 2 yes 2021-04-04T09:43:27.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent mirror pool info 2021-04-04T09:43:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parent mirror pool info 2021-04-04T09:43:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace mirror pool info 2021-04-04T09:43:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: image 2021-04-04T09:43:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Site Name: cluster2 2021-04-04T09:43:27.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Peer Sites: 2021-04-04T09:43:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:UUID: 66f7b743-d0ed-4180-9599-aa4c8b596b8d 2021-04-04T09:43:27.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Name: cluster1 2021-04-04T09:43:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirror UUID: 1cd21fe3-be3b-4a58-ac2d-a25de318debc 2021-04-04T09:43:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Direction: rx-tx 2021-04-04T09:43:27.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Client: client.mirror 2021-04-04T09:43:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:27.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent mirror pool status 2021-04-04T09:43:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parent mirror pool status 2021-04-04T09:43:27.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror_parent --namespace mirror pool status --verbose 2021-04-04T09:43:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: UNKNOWN 2021-04-04T09:43:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: UNKNOWN 2021-04-04T09:43:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: OK 2021-04-04T09:43:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 1 total 2021-04-04T09:43:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 1 replaying 2021-04-04T09:43:27.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2021-04-04T09:43:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: none 2021-04-04T09:43:27.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2021-04-04T09:43:27.726 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_parent: 2021-04-04T09:43:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: ed3379f2-3a26-4e23-8dd8-e2567a8361ce 2021-04-04T09:43:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T09:43:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T09:43:27.727 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T09:43:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T09:43:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T09:43:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T09:43:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":8},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":8}} 2021-04-04T09:43:27.728 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T09:43:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:27.731 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ls 2021-04-04T09:43:27.766 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_parent info 2021-04-04T09:43:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_parent info 2021-04-04T09:43:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace info test_parent 2021-04-04T09:43:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_parent': 2021-04-04T09:43:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T09:43:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T09:43:27.823 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2021-04-04T09:43:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 121aa1d3d42f 2021-04-04T09:43:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.121aa1d3d42f 2021-04-04T09:43:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T09:43:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T09:43:27.824 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T09:43:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T09:43:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:33:55 2021 2021-04-04T09:43:27.825 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:33:55 2021 2021-04-04T09:43:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:33:55 2021 2021-04-04T09:43:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 121aa1d3d42f 2021-04-04T09:43:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T09:43:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T09:43:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: ed3379f2-3a26-4e23-8dd8-e2567a8361ce 2021-04-04T09:43:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T09:43:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_parent journal status 2021-04-04T09:43:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_parent journal status 2021-04-04T09:43:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace journal status --image test_parent 2021-04-04T09:43:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T09:43:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T09:43:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T09:43:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=1, tag_tid=8, entry_tid=1], [object_number=0, tag_tid=8, entry_tid=0], [object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2]]], state=connected] 2021-04-04T09:43:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=1cd21fe3-be3b-4a58-ac2d-a25de318debc, commit_position=[positions=[[object_number=1, tag_tid=8, entry_tid=1], [object_number=0, tag_tid=8, entry_tid=0], [object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2]]], state=connected] 2021-04-04T09:43:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_parent snapshots 2021-04-04T09:43:27.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.899 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_parent snapshots 2021-04-04T09:43:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace snap ls --all test_parent 2021-04-04T09:43:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T09:43:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 4 snap 128 MiB yes Sun Apr 4 09:33:55 2021 user 2021-04-04T09:43:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent rbd_mirroring omap vals 2021-04-04T09:43:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parent rbd_mirroring omap vals 2021-04-04T09:43:27.959 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace listomapvals rbd_mirroring 2021-04-04T09:43:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_ed3379f2-3a26-4e23-8dd8-e2567a8361ce 2021-04-04T09:43:27.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T09:43:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 31 61 61 31 64 33 64 34 32 66 |....121aa1d3d42f| 2021-04-04T09:43:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T09:43:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_121aa1d3d42f 2021-04-04T09:43:27.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T09:43:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 64 33 33 37 39 |..*...$...ed3379| 2021-04-04T09:43:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 66 32 2d 33 61 32 36 2d 34 65 32 33 2d 38 64 64 |f2-3a26-4e23-8dd| 2021-04-04T09:43:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 38 2d 65 32 35 36 37 61 38 33 36 31 63 65 01 00 |8-e2567a8361ce..| 2021-04-04T09:43:27.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T09:43:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2021-04-04T09:43:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2021-04-04T09:43:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 00 00 00 |....| 2021-04-04T09:43:28.000 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2021-04-04T09:43:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_peer_66f7b743-d0ed-4180-9599-aa4c8b596b8d 2021-04-04T09:43:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (132 bytes) : 2021-04-04T09:43:28.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 36 36 66 37 62 37 |..~...$...66f7b7| 2021-04-04T09:43:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 34 33 2d 64 30 65 64 2d 34 31 38 30 2d 39 35 39 |43-d0ed-4180-959| 2021-04-04T09:43:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 39 2d 61 61 34 63 38 62 35 39 36 62 38 64 08 00 |9-aa4c8b596b8d..| 2021-04-04T09:43:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 63 6c 75 73 74 65 72 31 0d 00 00 00 63 6c |..cluster1....cl| 2021-04-04T09:43:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2021-04-04T09:43:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 ff ff ff 02 24 00 00 00 31 63 64 32 31 66 65 33 |....$...1cd21fe3| 2021-04-04T09:43:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2d 62 65 33 62 2d 34 61 35 38 2d 61 63 32 64 2d |-be3b-4a58-ac2d-| 2021-04-04T09:43:28.003 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 61 32 35 64 65 33 31 38 64 65 62 63 23 8a 69 60 |a25de318debc#.i`| 2021-04-04T09:43:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 9d 5a a1 21 |.Z.!| 2021-04-04T09:43:28.004 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000084 2021-04-04T09:43:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.005 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2021-04-04T09:43:28.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2021-04-04T09:43:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 64 65 35 33 36 36 32 39 2d 65 62 31 65 2d 34 64 |de536629-eb1e-4d| 2021-04-04T09:43:28.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 34 30 2d 62 61 64 33 2d 37 62 62 37 62 32 62 34 |40-bad3-7bb7b2b4| 2021-04-04T09:43:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 36 33 66 64 |63fd| 2021-04-04T09:43:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2021-04-04T09:43:28.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_ed3379f2-3a26-4e23-8dd8-e2567a8361ce_1cd21fe3-be3b-4a58-ac2d-a25de318debc 2021-04-04T09:43:28.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T09:43:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 f7 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T09:43:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 d5 45 04 51 10 00 |...........E.Q..| 2021-04-04T09:43:28.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T09:43:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T09:43:28.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T09:43:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T09:43:28.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T09:43:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T09:43:28.013 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T09:43:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T09:43:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T09:43:28.014 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 31 2c 22 6f 62 6a 65 63 |ry_tid":1,"objec| 2021-04-04T09:43:28.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 61 67 |t_number":1,"tag| 2021-04-04T09:43:28.015 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 38 7d 2c 22 70 72 69 6d 61 72 |_tid":8},"primar| 2021-04-04T09:43:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T09:43:28.016 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 31 2c 22 6f 62 6a 65 |try_tid":1,"obje| 2021-04-04T09:43:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 61 |ct_number":1,"ta| 2021-04-04T09:43:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 38 7d 7d 23 8a 69 60 8f 40 |g_tid":8}}#.i`.@| 2021-04-04T09:43:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 8c 10 00 |...| 2021-04-04T09:43:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T09:43:28.018 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent ns1 images 2021-04-04T09:43:28.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parent ns1 images 2021-04-04T09:43:28.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls -l 2021-04-04T09:43:28.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns1' does not exist. 2021-04-04T09:43:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: listing images failed: (2) No such file or directory 2021-04-04T09:43:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:28.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parentns1 mirror pool info 2021-04-04T09:43:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parentns1 mirror pool info 2021-04-04T09:43:28.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool info 2021-04-04T09:43:28.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns1' does not exist. 2021-04-04T09:43:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parentns1 mirror pool status 2021-04-04T09:43:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parentns1 mirror pool status 2021-04-04T09:43:28.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool status --verbose 2021-04-04T09:43:28.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns1' does not exist. 2021-04-04T09:43:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:28.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls 2021-04-04T09:43:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parent ns1 rbd_mirroring omap vals 2021-04-04T09:43:28.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent ns1 rbd_mirroring omap vals 2021-04-04T09:43:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2021-04-04T09:43:28.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2021-04-04T09:43:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent ns2 images 2021-04-04T09:43:28.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parent ns2 images 2021-04-04T09:43:28.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls -l 2021-04-04T09:43:28.236 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns2' does not exist. 2021-04-04T09:43:28.237 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: listing images failed: (2) No such file or directory 2021-04-04T09:43:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.240 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parentns2 mirror pool info 2021-04-04T09:43:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parentns2 mirror pool info 2021-04-04T09:43:28.242 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool info 2021-04-04T09:43:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns2' does not exist. 2021-04-04T09:43:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parentns2 mirror pool status 2021-04-04T09:43:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parentns2 mirror pool status 2021-04-04T09:43:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool status --verbose 2021-04-04T09:43:28.315 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns2' does not exist. 2021-04-04T09:43:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:28.319 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.320 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls 2021-04-04T09:43:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent ns2 rbd_mirroring omap vals 2021-04-04T09:43:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parent ns2 rbd_mirroring omap vals 2021-04-04T09:43:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2021-04-04T09:43:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2021-04-04T09:43:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:28.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret 2021-04-04T09:43:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T09:43:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:0 2021-04-04T09:43:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T09:43:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:43:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T09:43:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:43:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:43:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:43:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:43:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:43:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:43:28.403 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:43:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:43:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:43:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:43:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 pid file 2021-04-04T09:43:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2021-04-04T09:43:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2021-04-04T09:43:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ ! -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid ] 2021-04-04T09:43:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid 2021-04-04T09:43:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2021-04-04T09:43:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pid=34123 2021-04-04T09:43:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -z 34123 ] 2021-04-04T09:43:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo rbd-mirror process in ps output: 2021-04-04T09:43:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: rbd-mirror process in ps output: 2021-04-04T09:43:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:43:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=34123 NR == 1 {print} $2 == pid {print; exit 1} 2021-04-04T09:43:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 2021-04-04T09:43:28.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:ubuntu 34123 1.2 0.2 1032680 80468 ? Ssl 09:40 0:02 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:43:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:28.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T09:43:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T09:43:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:43:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:43:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T09:43:28.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T09:43:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:43:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:43:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:43:28.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:43:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T09:43:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:43:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:43:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T09:43:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T09:43:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T09:43:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:43:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T09:43:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ ! -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok ] 2021-04-04T09:43:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 rbd-mirror status 2021-04-04T09:43:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 rbd-mirror status 2021-04-04T09:43:28.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status 2021-04-04T09:43:28.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2021-04-04T09:43:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool_replayers": [ 2021-04-04T09:43:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:43:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "peer": "uuid: ba5e7ea1-e930-4550-b570-945e322d5ab4 cluster: cluster2 client: client.mirror", 2021-04-04T09:43:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool": "mirror", 2021-04-04T09:43:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4944", 2021-04-04T09:43:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "running", 2021-04-04T09:43:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader_instance_id": "4944", 2021-04-04T09:43:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader": true, 2021-04-04T09:43:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instances": [ 2021-04-04T09:43:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4944", 2021-04-04T09:43:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4948", 2021-04-04T09:43:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4957", 2021-04-04T09:43:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4961" 2021-04-04T09:43:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ], 2021-04-04T09:43:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.0.34123.cluster1.93932179281216.asok", 2021-04-04T09:43:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.34123.cluster2.93932191495488.asok", 2021-04-04T09:43:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "sync_throttler": { 2021-04-04T09:43:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 5, 2021-04-04T09:43:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:43:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:43:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "deletion_throttler": { 2021-04-04T09:43:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 1, 2021-04-04T09:43:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:43:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:43:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [ 2021-04-04T09:43:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:43:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "mirror/test_force_promote", 2021-04-04T09:43:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Stopped" 2021-04-04T09:43:28.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:43:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "mirror/test2", 2021-04-04T09:43:28.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:43:28.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:28.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:43:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "mirror/test_clone_v1", 2021-04-04T09:43:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:43:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:28.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:43:28.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "mirror/test1", 2021-04-04T09:43:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:43:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:28.577 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ], 2021-04-04T09:43:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter": { 2021-04-04T09:43:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter_status": { 2021-04-04T09:43:28.578 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "delete_images_queue": [], 2021-04-04T09:43:28.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "failed_deletes_queue": [] 2021-04-04T09:43:28.579 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "namespaces": [ 2021-04-04T09:43:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:43:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "ns1", 2021-04-04T09:43:28.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [], 2021-04-04T09:43:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter": { 2021-04-04T09:43:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter_status": { 2021-04-04T09:43:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "delete_images_queue": [], 2021-04-04T09:43:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "failed_deletes_queue": [] 2021-04-04T09:43:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:43:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "ns2", 2021-04-04T09:43:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [], 2021-04-04T09:43:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter": { 2021-04-04T09:43:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter_status": { 2021-04-04T09:43:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "delete_images_queue": [], 2021-04-04T09:43:28.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "failed_deletes_queue": [] 2021-04-04T09:43:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:28.587 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:28.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ] 2021-04-04T09:43:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:43:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "peer": "uuid: 062cbc91-4713-4ded-b744-6fda24ecf93a cluster: cluster2 client: client.mirror", 2021-04-04T09:43:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool": "mirror_parent", 2021-04-04T09:43:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4964", 2021-04-04T09:43:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "running", 2021-04-04T09:43:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader_instance_id": "4964", 2021-04-04T09:43:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader": true, 2021-04-04T09:43:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instances": [ 2021-04-04T09:43:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4964", 2021-04-04T09:43:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4971", 2021-04-04T09:43:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4975" 2021-04-04T09:43:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ], 2021-04-04T09:43:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.0.34123.cluster1.93932204488000.asok", 2021-04-04T09:43:28.594 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.34123.cluster2.93932216907072.asok", 2021-04-04T09:43:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "sync_throttler": { 2021-04-04T09:43:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 5, 2021-04-04T09:43:28.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:43:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:43:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:28.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "deletion_throttler": { 2021-04-04T09:43:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 1, 2021-04-04T09:43:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T09:43:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T09:43:28.597 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [ 2021-04-04T09:43:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T09:43:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "mirror_parent/test_parent", 2021-04-04T09:43:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T09:43:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ], 2021-04-04T09:43:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter": { 2021-04-04T09:43:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter_status": { 2021-04-04T09:43:28.599 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "delete_images_queue": [], 2021-04-04T09:43:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "failed_deletes_queue": [] 2021-04-04T09:43:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T09:43:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "namespaces": [] 2021-04-04T09:43:28.600 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T09:43:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ] 2021-04-04T09:43:28.601 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2021-04-04T09:43:28.602 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:28.605 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:1 2021-04-04T09:43:28.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T09:43:28.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:43:28.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T09:43:28.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:43:28.606 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T09:43:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:43:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:43:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:43:28.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:43:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T09:43:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:43:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:43:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:43:28.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 pid file 2021-04-04T09:43:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2021-04-04T09:43:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2021-04-04T09:43:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ ! -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid ] 2021-04-04T09:43:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid 2021-04-04T09:43:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2021-04-04T09:43:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pid=34151 2021-04-04T09:43:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -z 34151 ] 2021-04-04T09:43:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: rbd-mirror process in ps output: 2021-04-04T09:43:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo rbd-mirror process in ps output: 2021-04-04T09:43:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T09:43:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=34151 NR == 1 {print} $2 == pid {print; exit 1} 2021-04-04T09:43:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 2021-04-04T09:43:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:ubuntu 34151 0.4 0.1 794620 50612 ? Ssl 09:40 0:00 rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T09:43:28.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T09:43:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T09:43:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T09:43:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T09:43:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T09:43:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T09:43:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T09:43:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T09:43:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T09:43:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T09:43:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T09:43:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:43:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T09:43:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T09:43:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T09:43:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T09:43:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T09:43:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T09:43:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:43:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T09:43:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ ! -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok ] 2021-04-04T09:43:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 rbd-mirror status 2021-04-04T09:43:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 rbd-mirror status 2021-04-04T09:43:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status 2021-04-04T10:11:25.582 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T10:11:25.578+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c750 2021-04-04T10:26:21.452 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T10:26:21.447+0000 7f850200c700 -1 reset not still connected to 0x5605c1720f70 2021-04-04T10:26:21.452 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T10:26:21.447+0000 7f850200c700 -1 reset not still connected to 0x5605c1723e90 2021-04-04T10:41:46.241 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T10:41:46.233+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47e503b50 2021-04-04T10:45:59.292 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T10:45:59.287+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47ed31810 2021-04-04T10:45:59.293 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T10:45:59.287+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f728ff0 2021-04-04T10:45:59.293 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T10:45:59.287+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f729c20 2021-04-04T10:55:33.318 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2021-04-04T10:55:33.309+0000 7f6f409ce700 -1 reset not still connected to 0x55d2c8955a80 2021-04-04T10:57:14.313 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T10:57:14.307+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47ed31810 2021-04-04T10:57:14.313 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T10:57:14.307+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f728ff0 2021-04-04T10:57:14.314 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T10:57:14.307+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f729c20 2021-04-04T11:41:12.047 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T11:41:12.038+0000 7f850200c700 -1 reset not still connected to 0x5605c1720f70 2021-04-04T11:41:12.079 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T11:41:12.038+0000 7f850200c700 -1 reset not still connected to 0x5605c1723e90 2021-04-04T12:06:03.559 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T12:06:03.552+0000 7f850200c700 -1 reset not still connected to 0x5605c1720f70 2021-04-04T12:06:03.581 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T12:06:03.552+0000 7f850200c700 -1 reset not still connected to 0x5605c1723e90 2021-04-04T12:13:26.291 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T12:13:26.285+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89dad0 2021-04-04T12:30:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:Terminated 2021-04-04T12:30:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cleanup 143 2021-04-04T12:30:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local error_code=143 2021-04-04T12:30:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set +e 2021-04-04T12:30:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 143 -ne 0 ] 2021-04-04T12:30:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ status 2021-04-04T12:30:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster daemon image_pool image_ns image 2021-04-04T12:30:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 status 2021-04-04T12:30:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 status 2021-04-04T12:30:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster1 -s 2021-04-04T12:30:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: cluster: 2021-04-04T12:30:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 7fae7b04-7e84-4b67-8401-2f7942dda448 2021-04-04T12:30:34.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: HEALTH_OK 2021-04-04T12:30:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: services: 2021-04-04T12:30:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mon: 1 daemons, quorum a (age 3h) 2021-04-04T12:30:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mgr: x(active, since 3h) 2021-04-04T12:30:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: osd: 3 osds: 3 up (since 3h), 3 in (since 3h) 2021-04-04T12:30:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: rbd-mirror: 4 daemons active (1 hosts) 2021-04-04T12:30:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: data: 2021-04-04T12:30:34.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: pools: 4 pools, 137 pgs 2021-04-04T12:30:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: objects: 420 objects, 835 MiB 2021-04-04T12:30:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: usage: 2.6 GiB used, 265 GiB / 268 GiB avail 2021-04-04T12:30:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: pgs: 137 active+clean 2021-04-04T12:30:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: io: 2021-04-04T12:30:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client: 7.7 KiB/s rd, 682 B/s wr, 7 op/s rd, 0 op/s wr 2021-04-04T12:30:34.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster1 service dump 2021-04-04T12:30:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2021-04-04T12:30:34.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "epoch": 63, 2021-04-04T12:30:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "modified": "2021-04-04T12:30:17.726187+0000", 2021-04-04T12:30:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "services": { 2021-04-04T12:30:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "rbd-mirror": { 2021-04-04T12:30:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "daemons": { 2021-04-04T12:30:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "summary": "", 2021-04-04T12:30:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4934": { 2021-04-04T12:30:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_epoch": 11, 2021-04-04T12:30:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_stamp": "2021-04-04T09:40:29.087999+0000", 2021-04-04T12:30:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "gid": 4934, 2021-04-04T12:30:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "addr": "172.21.15.122:0/3613086818", 2021-04-04T12:30:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "metadata": { 2021-04-04T12:30:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "arch": "x86_64", 2021-04-04T12:30:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_release": "pacific", 2021-04-04T12:30:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version": "ceph version 16.2.0-22-gf4fb4141 (f4fb41419a42bee43a2069637ef17064073bf30b) pacific (stable)", 2021-04-04T12:30:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version_short": "16.2.0-22-gf4fb4141", 2021-04-04T12:30:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-04-04T12:30:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro": "ubuntu", 2021-04-04T12:30:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2021-04-04T12:30:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_version": "18.04", 2021-04-04T12:30:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "hostname": "smithi122", 2021-04-04T12:30:34.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "id": "mirror.0", 2021-04-04T12:30:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4934", 2021-04-04T12:30:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_description": "#144-Ubuntu SMP Fri Mar 19 14:12:35 UTC 2021", 2021-04-04T12:30:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_version": "4.15.0-140-generic", 2021-04-04T12:30:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_swap_kb": "2097148", 2021-04-04T12:30:34.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_total_kb": "32823744", 2021-04-04T12:30:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "os": "Linux" 2021-04-04T12:30:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "task_status": {} 2021-04-04T12:30:34.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4938": { 2021-04-04T12:30:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_epoch": 11, 2021-04-04T12:30:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_stamp": "2021-04-04T09:40:29.104200+0000", 2021-04-04T12:30:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "gid": 4938, 2021-04-04T12:30:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "addr": "172.21.15.122:0/75107711", 2021-04-04T12:30:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "metadata": { 2021-04-04T12:30:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "arch": "x86_64", 2021-04-04T12:30:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_release": "pacific", 2021-04-04T12:30:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version": "ceph version 16.2.0-22-gf4fb4141 (f4fb41419a42bee43a2069637ef17064073bf30b) pacific (stable)", 2021-04-04T12:30:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version_short": "16.2.0-22-gf4fb4141", 2021-04-04T12:30:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-04-04T12:30:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro": "ubuntu", 2021-04-04T12:30:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2021-04-04T12:30:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_version": "18.04", 2021-04-04T12:30:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "hostname": "smithi122", 2021-04-04T12:30:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "id": "mirror.1", 2021-04-04T12:30:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4938", 2021-04-04T12:30:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_description": "#144-Ubuntu SMP Fri Mar 19 14:12:35 UTC 2021", 2021-04-04T12:30:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_version": "4.15.0-140-generic", 2021-04-04T12:30:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_swap_kb": "2097148", 2021-04-04T12:30:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_total_kb": "32823744", 2021-04-04T12:30:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "os": "Linux" 2021-04-04T12:30:34.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "task_status": {} 2021-04-04T12:30:34.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4945": { 2021-04-04T12:30:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_epoch": 11, 2021-04-04T12:30:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_stamp": "2021-04-04T09:40:29.141998+0000", 2021-04-04T12:30:34.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "gid": 4945, 2021-04-04T12:30:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "addr": "172.21.15.122:0/4031853056", 2021-04-04T12:30:34.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "metadata": { 2021-04-04T12:30:34.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "arch": "x86_64", 2021-04-04T12:30:34.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_release": "pacific", 2021-04-04T12:30:34.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version": "ceph version 16.2.0-22-gf4fb4141 (f4fb41419a42bee43a2069637ef17064073bf30b) pacific (stable)", 2021-04-04T12:30:34.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version_short": "16.2.0-22-gf4fb4141", 2021-04-04T12:30:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-04-04T12:30:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro": "ubuntu", 2021-04-04T12:30:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2021-04-04T12:30:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_version": "18.04", 2021-04-04T12:30:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "hostname": "smithi122", 2021-04-04T12:30:34.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "id": "mirror.2", 2021-04-04T12:30:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4945", 2021-04-04T12:30:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_description": "#144-Ubuntu SMP Fri Mar 19 14:12:35 UTC 2021", 2021-04-04T12:30:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_version": "4.15.0-140-generic", 2021-04-04T12:30:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_swap_kb": "2097148", 2021-04-04T12:30:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_total_kb": "32823744", 2021-04-04T12:30:34.703 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "os": "Linux" 2021-04-04T12:30:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "task_status": {} 2021-04-04T12:30:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:34.704 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4952": { 2021-04-04T12:30:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_epoch": 11, 2021-04-04T12:30:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "start_stamp": "2021-04-04T09:40:29.179687+0000", 2021-04-04T12:30:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "gid": 4952, 2021-04-04T12:30:34.705 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "addr": "172.21.15.122:0/1332119204", 2021-04-04T12:30:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "metadata": { 2021-04-04T12:30:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "arch": "x86_64", 2021-04-04T12:30:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_release": "pacific", 2021-04-04T12:30:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version": "ceph version 16.2.0-22-gf4fb4141 (f4fb41419a42bee43a2069637ef17064073bf30b) pacific (stable)", 2021-04-04T12:30:34.706 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "ceph_version_short": "16.2.0-22-gf4fb4141", 2021-04-04T12:30:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-04-04T12:30:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro": "ubuntu", 2021-04-04T12:30:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2021-04-04T12:30:34.707 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "distro_version": "18.04", 2021-04-04T12:30:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "hostname": "smithi122", 2021-04-04T12:30:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "id": "mirror.3", 2021-04-04T12:30:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4952", 2021-04-04T12:30:34.708 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_description": "#144-Ubuntu SMP Fri Mar 19 14:12:35 UTC 2021", 2021-04-04T12:30:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "kernel_version": "4.15.0-140-generic", 2021-04-04T12:30:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_swap_kb": "2097148", 2021-04-04T12:30:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "mem_total_kb": "32823744", 2021-04-04T12:30:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "os": "Linux" 2021-04-04T12:30:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "task_status": {} 2021-04-04T12:30:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:34.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2021-04-04T12:30:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster1 service status 2021-04-04T12:30:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2021-04-04T12:30:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "rbd-mirror": { 2021-04-04T12:30:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4934": { 2021-04-04T12:30:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status_stamp": "2021-04-04T12:30:34.227197+0000", 2021-04-04T12:30:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "last_beacon": "2021-04-04T12:30:34.227197+0000", 2021-04-04T12:30:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status": { 2021-04-04T12:30:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":4,\"image_error_count\":0,\"image_local_count\":10,\"image_remote_count\":9,\"image_warning_count\":0,\"instance_id\":\"4944\",\"leader\":true,\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":2,\"image_remote_count\":1,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":1,\"image_remote_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":1,\"image_error_count\":0,\"image_local_count\":1,\"image_remote_count\":1,\"image_warning_count\":0,\"instance_id\":\"4964\",\"leader\":true}}" 2021-04-04T12:30:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4938": { 2021-04-04T12:30:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status_stamp": "2021-04-04T12:24:04.426018+0000", 2021-04-04T12:30:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "last_beacon": "2021-04-04T12:30:34.442037+0000", 2021-04-04T12:30:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status": { 2021-04-04T12:30:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":4,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4948\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":1,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{}}}" 2021-04-04T12:30:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4945": { 2021-04-04T12:30:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status_stamp": "2021-04-04T12:30:34.246190+0000", 2021-04-04T12:30:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "last_beacon": "2021-04-04T12:30:34.246190+0000", 2021-04-04T12:30:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status": { 2021-04-04T12:30:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":1,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4957\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4975\"}}" 2021-04-04T12:30:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4952": { 2021-04-04T12:30:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status_stamp": "2021-04-04T12:30:34.546790+0000", 2021-04-04T12:30:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "last_beacon": "2021-04-04T12:30:34.546790+0000", 2021-04-04T12:30:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "status": { 2021-04-04T12:30:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4961\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"4971\"}}" 2021-04-04T12:30:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2021-04-04T12:30:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror images 2021-04-04T12:30:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror images 2021-04-04T12:30:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ls -l 2021-04-04T12:30:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:NAME SIZE PARENT FMT PROT LOCK 2021-04-04T12:30:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test 128 MiB 2 excl 2021-04-04T12:30:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test1 128 MiB 2 excl 2021-04-04T12:30:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2 128 MiB 2 excl 2021-04-04T12:30:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2@snap1 128 MiB 2 2021-04-04T12:30:35.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2@snap2 128 MiB 2 2021-04-04T12:30:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2@snap_rename_20 128 MiB 2 2021-04-04T12:30:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test4 128 MiB 2 excl 2021-04-04T12:30:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test4@snap1 128 MiB 2 yes 2021-04-04T12:30:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test4@snap2 128 MiB 2 yes 2021-04-04T12:30:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone 128 MiB mirror_parent/test_parent@snap 2 excl 2021-04-04T12:30:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone1 128 MiB mirror_parent/test_parent@snap 2 2021-04-04T12:30:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone_v1 128 MiB mirror_parent/test_parent@snap 2 excl 2021-04-04T12:30:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool 128 MiB 2 excl 2021-04-04T12:30:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool@snap1 128 MiB 2 2021-04-04T12:30:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool@snap2 128 MiB 2 2021-04-04T12:30:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_force_promote 128 MiB 2 2021-04-04T12:30:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror mirror pool info 2021-04-04T12:30:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror mirror pool info 2021-04-04T12:30:35.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace mirror pool info 2021-04-04T12:30:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: pool 2021-04-04T12:30:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Site Name: cluster1 2021-04-04T12:30:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Peer Sites: 2021-04-04T12:30:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:UUID: ba5e7ea1-e930-4550-b570-945e322d5ab4 2021-04-04T12:30:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Name: cluster2 2021-04-04T12:30:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirror UUID: f36635b8-4737-4365-816f-3d9690ce95a7 2021-04-04T12:30:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Direction: rx-tx 2021-04-04T12:30:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Client: client.mirror 2021-04-04T12:30:35.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror mirror pool status 2021-04-04T12:30:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror mirror pool status 2021-04-04T12:30:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror --namespace mirror pool status --verbose 2021-04-04T12:30:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: WARNING 2021-04-04T12:30:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: OK 2021-04-04T12:30:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: WARNING 2021-04-04T12:30:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 9 total 2021-04-04T12:30:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 9 unknown 2021-04-04T12:30:35.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2021-04-04T12:30:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4934: 2021-04-04T12:30:35.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4944 2021-04-04T12:30:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.0 2021-04-04T12:30:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T12:30:35.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T12:30:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: true 2021-04-04T12:30:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T12:30:35.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4938: 2021-04-04T12:30:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4948 2021-04-04T12:30:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.1 2021-04-04T12:30:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T12:30:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T12:30:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T12:30:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T12:30:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4945: 2021-04-04T12:30:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4957 2021-04-04T12:30:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.2 2021-04-04T12:30:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T12:30:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T12:30:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T12:30:35.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T12:30:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4952: 2021-04-04T12:30:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4961 2021-04-04T12:30:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.3 2021-04-04T12:30:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T12:30:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T12:30:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T12:30:35.281 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T12:30:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2021-04-04T12:30:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test: 2021-04-04T12:30:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 3c8112b2-6f20-4a55-833f-e3a0d1e92beb 2021-04-04T12:30:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T12:30:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.2 on smithi122 2021-04-04T12:30:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 12:30:29 2021-04-04T12:30:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T12:30:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:35.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test1: 2021-04-04T12:30:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T12:30:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T12:30:35.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T12:30:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 12:30:29 2021-04-04T12:30:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T12:30:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2: 2021-04-04T12:30:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 80ed7469-0640-4a03-9952-a34e4d433832 2021-04-04T12:30:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":22},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":22}} 2021-04-04T12:30:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T12:30:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 12:30:29 2021-04-04T12:30:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T12:30:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test4: 2021-04-04T12:30:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 698bca8d-5596-41e0-b6a2-71fc39245e79 2021-04-04T12:30:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T12:30:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T12:30:35.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:43:29 2021-04-04T12:30:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T12:30:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:35.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone: 2021-04-04T12:30:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: a9fc372c-84b2-41cd-a172-f5e0c63e6d90 2021-04-04T12:30:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T12:30:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T12:30:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:43:29 2021-04-04T12:30:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T12:30:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone1: 2021-04-04T12:30:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 14c06da4-7ebf-41e7-8aa7-0b0a1328684a 2021-04-04T12:30:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T12:30:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: local image is primary 2021-04-04T12:30:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T12:30:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:43:24 2021-04-04T12:30:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T12:30:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone_v1: 2021-04-04T12:30:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: caa231b2-f85a-456d-b241-95227b74117e 2021-04-04T12:30:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T12:30:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T12:30:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 12:30:29 2021-04-04T12:30:35.555 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T12:30:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:35.556 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool: 2021-04-04T12:30:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 63d2ca14-0fae-4993-96ee-cffbec8f3ffc 2021-04-04T12:30:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T12:30:35.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T12:30:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:43:29 2021-04-04T12:30:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T12:30:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_force_promote: 2021-04-04T12:30:35.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T12:30:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T12:30:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: local image is primary 2021-04-04T12:30:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T12:30:35.561 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 12:30:34 2021-04-04T12:30:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T12:30:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:35.562 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:35.564 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ls 2021-04-04T12:30:35.595 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test info 2021-04-04T12:30:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test info 2021-04-04T12:30:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test 2021-04-04T12:30:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2021-04-04T12:30:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T12:30:35.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 135d95a808e2 2021-04-04T12:30:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.135d95a808e2 2021-04-04T12:30:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:35.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:40:43 2021 2021-04-04T12:30:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:40:43 2021 2021-04-04T12:30:35.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:40:43 2021 2021-04-04T12:30:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 135d95a808e2 2021-04-04T12:30:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:35.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 3c8112b2-6f20-4a55-833f-e3a0d1e92beb 2021-04-04T12:30:35.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T12:30:35.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test journal status 2021-04-04T12:30:35.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test journal status 2021-04-04T12:30:35.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test 2021-04-04T12:30:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:35.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:35.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test snapshots 2021-04-04T12:30:35.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:35.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test snapshots 2021-04-04T12:30:35.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test 2021-04-04T12:30:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test1 info 2021-04-04T12:30:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test1 info 2021-04-04T12:30:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test1 2021-04-04T12:30:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test1': 2021-04-04T12:30:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T12:30:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10dd683f3cad 2021-04-04T12:30:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10dd683f3cad 2021-04-04T12:30:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:31:07 2021 2021-04-04T12:30:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:31:07 2021 2021-04-04T12:30:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:31:07 2021 2021-04-04T12:30:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10dd683f3cad 2021-04-04T12:30:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T12:30:36.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T12:30:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test1 journal status 2021-04-04T12:30:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test1 journal status 2021-04-04T12:30:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test1 2021-04-04T12:30:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:36.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:36.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:36.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:36.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test1 snapshots 2021-04-04T12:30:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test1 snapshots 2021-04-04T12:30:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test1 2021-04-04T12:30:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test2 info 2021-04-04T12:30:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:36.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test2 info 2021-04-04T12:30:36.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test2 2021-04-04T12:30:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test2': 2021-04-04T12:30:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 3 2021-04-04T12:30:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e0b93cfe9e 2021-04-04T12:30:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e0b93cfe9e 2021-04-04T12:30:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:35:24 2021 2021-04-04T12:30:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:35:24 2021 2021-04-04T12:30:36.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:35:24 2021 2021-04-04T12:30:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10e0b93cfe9e 2021-04-04T12:30:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 80ed7469-0640-4a03-9952-a34e4d433832 2021-04-04T12:30:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T12:30:36.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:36.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test2 journal status 2021-04-04T12:30:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test2 journal status 2021-04-04T12:30:36.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test2 2021-04-04T12:30:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:36.374 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test2 snapshots 2021-04-04T12:30:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test2 snapshots 2021-04-04T12:30:36.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test2 2021-04-04T12:30:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T12:30:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 44 snap1 128 MiB Sun Apr 4 09:35:25 2021 user 2021-04-04T12:30:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 45 snap2 128 MiB Sun Apr 4 09:35:25 2021 user 2021-04-04T12:30:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 52 snap_rename_20 128 MiB Sun Apr 4 09:39:55 2021 user 2021-04-04T12:30:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:36.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test4 info 2021-04-04T12:30:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test4 info 2021-04-04T12:30:36.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test4 2021-04-04T12:30:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test4': 2021-04-04T12:30:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2021-04-04T12:30:36.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e78fad30ae 2021-04-04T12:30:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e78fad30ae 2021-04-04T12:30:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:35:31 2021 2021-04-04T12:30:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:35:31 2021 2021-04-04T12:30:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:35:31 2021 2021-04-04T12:30:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10e78fad30ae 2021-04-04T12:30:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:36.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 698bca8d-5596-41e0-b6a2-71fc39245e79 2021-04-04T12:30:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T12:30:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:36.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test4 journal status 2021-04-04T12:30:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:36.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test4 journal status 2021-04-04T12:30:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test4 2021-04-04T12:30:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test4 snapshots 2021-04-04T12:30:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test4 snapshots 2021-04-04T12:30:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test4 2021-04-04T12:30:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T12:30:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 48 snap1 128 MiB yes Sun Apr 4 09:35:32 2021 user 2021-04-04T12:30:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 49 snap2 128 MiB yes Sun Apr 4 09:35:33 2021 user 2021-04-04T12:30:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone info 2021-04-04T12:30:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone info 2021-04-04T12:30:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test_clone 2021-04-04T12:30:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_clone': 2021-04-04T12:30:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T12:30:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ddbd07e7b5 2021-04-04T12:30:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ddbd07e7b5 2021-04-04T12:30:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:05 2021 2021-04-04T12:30:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:05 2021 2021-04-04T12:30:36.807 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:05 2021 2021-04-04T12:30:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: parent: mirror_parent/test_parent@snap 2021-04-04T12:30:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: overlap: 128 MiB 2021-04-04T12:30:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10ddbd07e7b5 2021-04-04T12:30:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: a9fc372c-84b2-41cd-a172-f5e0c63e6d90 2021-04-04T12:30:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T12:30:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone journal status 2021-04-04T12:30:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone journal status 2021-04-04T12:30:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test_clone 2021-04-04T12:30:36.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:36.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:36.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:36.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:36.905 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone snapshots 2021-04-04T12:30:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone snapshots 2021-04-04T12:30:36.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test_clone 2021-04-04T12:30:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone1 info 2021-04-04T12:30:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone1 info 2021-04-04T12:30:36.999 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test_clone1 2021-04-04T12:30:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_clone1': 2021-04-04T12:30:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:37.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T12:30:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12487c3dac7b 2021-04-04T12:30:37.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12487c3dac7b 2021-04-04T12:30:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:37.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:12 2021 2021-04-04T12:30:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:12 2021 2021-04-04T12:30:37.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:12 2021 2021-04-04T12:30:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: parent: mirror_parent/test_parent@snap 2021-04-04T12:30:37.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: overlap: 128 MiB 2021-04-04T12:30:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 12487c3dac7b 2021-04-04T12:30:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:37.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:37.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 14c06da4-7ebf-41e7-8aa7-0b0a1328684a 2021-04-04T12:30:37.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T12:30:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone1 journal status 2021-04-04T12:30:37.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone1 journal status 2021-04-04T12:30:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test_clone1 2021-04-04T12:30:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T12:30:37.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T12:30:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:37.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone1 snapshots 2021-04-04T12:30:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone1 snapshots 2021-04-04T12:30:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test_clone1 2021-04-04T12:30:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:37.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone_v1 info 2021-04-04T12:30:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone_v1 info 2021-04-04T12:30:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test_clone_v1 2021-04-04T12:30:37.360 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_clone_v1': 2021-04-04T12:30:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:37.361 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T12:30:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e71c96086d 2021-04-04T12:30:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e71c96086d 2021-04-04T12:30:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:37.363 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:37.364 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:14 2021 2021-04-04T12:30:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:14 2021 2021-04-04T12:30:37.365 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:14 2021 2021-04-04T12:30:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: parent: mirror_parent/test_parent@snap 2021-04-04T12:30:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: overlap: 128 MiB 2021-04-04T12:30:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10e71c96086d 2021-04-04T12:30:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:37.367 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: caa231b2-f85a-456d-b241-95227b74117e 2021-04-04T12:30:37.368 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T12:30:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone_v1 journal status 2021-04-04T12:30:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone_v1 journal status 2021-04-04T12:30:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test_clone_v1 2021-04-04T12:30:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:37.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:37.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone_v1 snapshots 2021-04-04T12:30:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone_v1 snapshots 2021-04-04T12:30:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test_clone_v1 2021-04-04T12:30:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool info 2021-04-04T12:30:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool info 2021-04-04T12:30:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test_data_pool 2021-04-04T12:30:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_data_pool': 2021-04-04T12:30:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2021-04-04T12:30:37.616 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ddcfb530aa 2021-04-04T12:30:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: data_pool: mirror_parent 2021-04-04T12:30:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.3.10ddcfb530aa 2021-04-04T12:30:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling, data-pool 2021-04-04T12:30:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:39 2021 2021-04-04T12:30:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:39 2021 2021-04-04T12:30:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:39 2021 2021-04-04T12:30:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10ddcfb530aa 2021-04-04T12:30:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 63d2ca14-0fae-4993-96ee-cffbec8f3ffc 2021-04-04T12:30:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T12:30:37.621 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool journal status 2021-04-04T12:30:37.623 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool journal status 2021-04-04T12:30:37.624 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test_data_pool 2021-04-04T12:30:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:37.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool snapshots 2021-04-04T12:30:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool snapshots 2021-04-04T12:30:37.699 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test_data_pool 2021-04-04T12:30:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T12:30:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 14 snap1 128 MiB Sun Apr 4 09:34:44 2021 user 2021-04-04T12:30:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 15 snap2 128 MiB Sun Apr 4 09:34:46 2021 user 2021-04-04T12:30:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:37.769 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_force_promote info 2021-04-04T12:30:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_force_promote info 2021-04-04T12:30:37.770 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace info test_force_promote 2021-04-04T12:30:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_force_promote': 2021-04-04T12:30:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T12:30:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ea33c092e 2021-04-04T12:30:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ea33c092e 2021-04-04T12:30:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:37.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:37.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:37.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:37.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:33:43 2021 2021-04-04T12:30:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:33:43 2021 2021-04-04T12:30:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:33:43 2021 2021-04-04T12:30:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10ea33c092e 2021-04-04T12:30:37.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T12:30:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T12:30:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_force_promote journal status 2021-04-04T12:30:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_force_promote journal status 2021-04-04T12:30:37.839 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace journal status --image test_force_promote 2021-04-04T12:30:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=101], [object_number=0, tag_tid=3, entry_tid=100], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected] 2021-04-04T12:30:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=f36635b8-4737-4365-816f-3d9690ce95a7, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_force_promote snapshots 2021-04-04T12:30:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_force_promote snapshots 2021-04-04T12:30:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace snap ls --all test_force_promote 2021-04-04T12:30:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror rbd_mirroring omap vals 2021-04-04T12:30:37.977 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror rbd_mirroring omap vals 2021-04-04T12:30:37.978 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror --namespace listomapvals rbd_mirroring 2021-04-04T12:30:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_14c06da4-7ebf-41e7-8aa7-0b0a1328684a 2021-04-04T12:30:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 34 38 37 63 33 64 61 63 37 62 |....12487c3dac7b| 2021-04-04T12:30:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_3c8112b2-6f20-4a55-833f-e3a0d1e92beb 2021-04-04T12:30:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 33 35 64 39 35 61 38 30 38 65 32 |....135d95a808e2| 2021-04-04T12:30:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T12:30:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (15 bytes) : 2021-04-04T12:30:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0b 00 00 00 31 30 65 61 33 33 63 30 39 32 65 |....10ea33c092e| 2021-04-04T12:30:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000000f 2021-04-04T12:30:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_63d2ca14-0fae-4993-96ee-cffbec8f3ffc 2021-04-04T12:30:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 64 64 63 66 62 35 33 30 61 61 |....10ddcfb530aa| 2021-04-04T12:30:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_698bca8d-5596-41e0-b6a2-71fc39245e79 2021-04-04T12:30:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 37 38 66 61 64 33 30 61 65 |....10e78fad30ae| 2021-04-04T12:30:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_80ed7469-0640-4a03-9952-a34e4d433832 2021-04-04T12:30:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 30 62 39 33 63 66 65 39 65 |....10e0b93cfe9e| 2021-04-04T12:30:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_a9fc372c-84b2-41cd-a172-f5e0c63e6d90 2021-04-04T12:30:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 64 64 62 64 30 37 65 37 62 35 |....10ddbd07e7b5| 2021-04-04T12:30:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_caa231b2-f85a-456d-b241-95227b74117e 2021-04-04T12:30:38.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 37 31 63 39 36 30 38 36 64 |....10e71c96086d| 2021-04-04T12:30:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:38.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T12:30:38.031 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 64 64 36 38 33 66 33 63 61 64 |....10dd683f3cad| 2021-04-04T12:30:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10dd683f3cad 2021-04-04T12:30:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 31 61 62 65 35 |..*...$...e1abe5| 2021-04-04T12:30:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 63 35 2d 61 37 37 33 2d 34 39 65 34 2d 39 34 30 |c5-a773-49e4-940| 2021-04-04T12:30:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 39 2d 62 32 62 66 37 39 63 66 64 65 33 31 01 00 |9-b2bf79cfde31..| 2021-04-04T12:30:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ddbd07e7b5 2021-04-04T12:30:38.034 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 61 39 66 63 33 37 |..*...$...a9fc37| 2021-04-04T12:30:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 32 63 2d 38 34 62 32 2d 34 31 63 64 2d 61 31 37 |2c-84b2-41cd-a17| 2021-04-04T12:30:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 2d 66 35 65 30 63 36 33 65 36 64 39 30 01 00 |2-f5e0c63e6d90..| 2021-04-04T12:30:38.035 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ddcfb530aa 2021-04-04T12:30:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 33 64 32 63 61 |..*...$...63d2ca| 2021-04-04T12:30:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 31 34 2d 30 66 61 65 2d 34 39 39 33 2d 39 36 65 |14-0fae-4993-96e| 2021-04-04T12:30:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 65 2d 63 66 66 62 65 63 38 66 33 66 66 63 01 00 |e-cffbec8f3ffc..| 2021-04-04T12:30:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e0b93cfe9e 2021-04-04T12:30:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 30 65 64 37 34 |..*...$...80ed74| 2021-04-04T12:30:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 36 39 2d 30 36 34 30 2d 34 61 30 33 2d 39 39 35 |69-0640-4a03-995| 2021-04-04T12:30:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 2d 61 33 34 65 34 64 34 33 33 38 33 32 01 00 |2-a34e4d433832..| 2021-04-04T12:30:38.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e71c96086d 2021-04-04T12:30:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 61 61 32 33 31 |..*...$...caa231| 2021-04-04T12:30:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 62 32 2d 66 38 35 61 2d 34 35 36 64 2d 62 32 34 |b2-f85a-456d-b24| 2021-04-04T12:30:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 31 2d 39 35 32 32 37 62 37 34 31 31 37 65 01 00 |1-95227b74117e..| 2021-04-04T12:30:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e78fad30ae 2021-04-04T12:30:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 39 38 62 63 61 |..*...$...698bca| 2021-04-04T12:30:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 64 2d 35 35 39 36 2d 34 31 65 30 2d 62 36 61 |8d-5596-41e0-b6a| 2021-04-04T12:30:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 2d 37 31 66 63 33 39 32 34 35 65 37 39 01 00 |2-71fc39245e79..| 2021-04-04T12:30:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ea33c092e 2021-04-04T12:30:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 33 39 65 37 36 |..*...$...439e76| 2021-04-04T12:30:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 66 37 2d 65 39 63 62 2d 34 66 39 32 2d 38 31 30 |f7-e9cb-4f92-810| 2021-04-04T12:30:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 64 2d 63 66 37 65 39 35 33 30 33 64 37 39 01 00 |d-cf7e95303d79..| 2021-04-04T12:30:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12487c3dac7b 2021-04-04T12:30:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 34 63 30 36 64 |..*...$...14c06d| 2021-04-04T12:30:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 61 34 2d 37 65 62 66 2d 34 31 65 37 2d 38 61 61 |a4-7ebf-41e7-8aa| 2021-04-04T12:30:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 37 2d 30 62 30 61 31 33 32 38 36 38 34 61 01 00 |7-0b0a1328684a..| 2021-04-04T12:30:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_135d95a808e2 2021-04-04T12:30:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:38.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 33 63 38 31 31 32 |..*...$...3c8112| 2021-04-04T12:30:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 62 32 2d 36 66 32 30 2d 34 61 35 35 2d 38 33 33 |b2-6f20-4a55-833| 2021-04-04T12:30:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 66 2d 65 33 61 30 64 31 65 39 32 62 65 62 01 00 |f-e3a0d1e92beb..| 2021-04-04T12:30:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2021-04-04T12:30:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2021-04-04T12:30:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 00 00 00 |....| 2021-04-04T12:30:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2021-04-04T12:30:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_peer_ba5e7ea1-e930-4550-b570-945e322d5ab4 2021-04-04T12:30:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (132 bytes) : 2021-04-04T12:30:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 62 61 35 65 37 65 |..~...$...ba5e7e| 2021-04-04T12:30:38.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 61 31 2d 65 39 33 30 2d 34 35 35 30 2d 62 35 37 |a1-e930-4550-b57| 2021-04-04T12:30:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 30 2d 39 34 35 65 33 32 32 64 35 61 62 34 08 00 |0-945e322d5ab4..| 2021-04-04T12:30:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2021-04-04T12:30:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2021-04-04T12:30:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 ff ff ff 02 24 00 00 00 66 33 36 36 33 35 62 38 |....$...f36635b8| 2021-04-04T12:30:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2d 34 37 33 37 2d 34 33 36 35 2d 38 31 36 66 2d |-4737-4365-816f-| 2021-04-04T12:30:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 33 64 39 36 39 30 63 65 39 35 61 37 8b 89 69 60 |3d9690ce95a7..i`| 2021-04-04T12:30:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 ec 6e 99 2d |.n.-| 2021-04-04T12:30:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000084 2021-04-04T12:30:38.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2021-04-04T12:30:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2021-04-04T12:30:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 34 65 63 62 63 64 32 37 2d 64 33 35 31 2d 34 34 |4ecbcd27-d351-44| 2021-04-04T12:30:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 35 30 2d 39 33 32 36 2d 32 37 61 39 36 66 33 35 |50-9326-27a96f35| 2021-04-04T12:30:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 34 39 33 |2493| 2021-04-04T12:30:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2021-04-04T12:30:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_09a6fbfb-f68f-4e71-98d3-0b0c50ff9d74 2021-04-04T12:30:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T12:30:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T12:30:38.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 f5 d2 c9 0d 00 | replay..i`.....| 2021-04-04T12:30:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_14c06da4-7ebf-41e7-8aa7-0b0a1328684a 2021-04-04T12:30:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (92 bytes) : 2021-04-04T12:30:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T12:30:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T12:30:38.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2021-04-04T12:30:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2021-04-04T12:30:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 61 72 79 3c 8a 69 60 e5 10 bc 0c 00 |ary<.i`.....| 2021-04-04T12:30:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000005c 2021-04-04T12:30:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_19dd5345-25d8-4554-8034-6c31e0df778e 2021-04-04T12:30:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T12:30:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T12:30:38.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 d5 4a 41 0e 00 | replay..i`.JA..| 2021-04-04T12:30:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_3c8112b2-6f20-4a55-833f-e3a0d1e92beb 2021-04-04T12:30:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T12:30:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 5d 13 00 00 00 00 00 00 01 |..,....]........| 2021-04-04T12:30:38.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2d 21 8d f0 10 00 |..........-!....| 2021-04-04T12:30:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:38.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T12:30:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T12:30:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T12:30:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T12:30:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T12:30:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T12:30:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 31 7d 2c 22 70 72 69 6d 61 72 |_tid":1},"primar| 2021-04-04T12:30:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T12:30:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T12:30:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T12:30:38.067 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 31 7d 7d 65 b1 69 60 79 76 |g_tid":1}}e.i`yv| 2021-04-04T12:30:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 83 10 00 |...| 2021-04-04T12:30:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T12:30:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_409477f3-cbd2-4eff-9e17-c87e509f4e40 2021-04-04T12:30:38.068 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T12:30:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T12:30:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 ee 85 fb 0d 00 | replay..i`.....| 2021-04-04T12:30:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T12:30:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (92 bytes) : 2021-04-04T12:30:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T12:30:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T12:30:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2021-04-04T12:30:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2021-04-04T12:30:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 61 72 79 6a b1 69 60 4b 84 c9 1a 00 |aryj.i`K....| 2021-04-04T12:30:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000005c 2021-04-04T12:30:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_63d2ca14-0fae-4993-96ee-cffbec8f3ffc 2021-04-04T12:30:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (295 bytes) : 2021-04-04T12:30:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T12:30:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T12:30:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 ef 00 00 00 04 e1 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T12:30:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T12:30:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T12:30:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T12:30:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 31 30 31 2c 22 6f 62 6a |ry_tid":101,"obj| 2021-04-04T12:30:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 65 63 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 |ect_number":1,"t| 2021-04-04T12:30:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 61 67 5f 74 69 64 22 3a 35 7d 2c 22 70 72 69 6d |ag_tid":5},"prim| 2021-04-04T12:30:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 72 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 |ary_position":{"| 2021-04-04T12:30:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 65 6e 74 72 79 5f 74 69 64 22 3a 31 30 31 2c 22 |entry_tid":101,"| 2021-04-04T12:30:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 6f 62 6a 65 63 74 5f 6e 75 6d 62 65 72 22 3a 31 |object_number":1| 2021-04-04T12:30:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 2c 22 74 61 67 5f 74 69 64 22 3a 35 7d 7d 41 8a |,"tag_tid":5}}A.| 2021-04-04T12:30:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 69 60 49 3e 30 0b 00 |i`I>0..| 2021-04-04T12:30:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000127 2021-04-04T12:30:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_690ce498-199d-4023-9c64-ff4d7599e591 2021-04-04T12:30:38.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T12:30:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T12:30:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 31 a7 41 0e 00 | replay..i`1.A..| 2021-04-04T12:30:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_698bca8d-5596-41e0-b6a2-71fc39245e79 2021-04-04T12:30:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T12:30:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T12:30:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T12:30:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:38.082 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T12:30:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T12:30:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T12:30:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T12:30:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T12:30:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T12:30:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 31 7d 2c 22 70 72 69 6d 61 72 |_tid":1},"primar| 2021-04-04T12:30:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T12:30:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T12:30:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T12:30:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 31 7d 7d 41 8a 69 60 9e 68 |g_tid":1}}A.i`.h| 2021-04-04T12:30:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 31 0b 00 |1..| 2021-04-04T12:30:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T12:30:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_80ed7469-0640-4a03-9952-a34e4d433832 2021-04-04T12:30:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (293 bytes) : 2021-04-04T12:30:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T12:30:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T12:30:38.086 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 ed 00 00 00 04 df 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T12:30:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T12:30:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T12:30:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T12:30:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 31 2c 22 6f 62 6a 65 63 |ry_tid":1,"objec| 2021-04-04T12:30:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 61 67 |t_number":1,"tag| 2021-04-04T12:30:38.089 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 32 32 7d 2c 22 70 72 69 6d 61 |_tid":22},"prima| 2021-04-04T12:30:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 72 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 |ry_position":{"e| 2021-04-04T12:30:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 6e 74 72 79 5f 74 69 64 22 3a 31 2c 22 6f 62 6a |ntry_tid":1,"obj| 2021-04-04T12:30:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 65 63 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 |ect_number":1,"t| 2021-04-04T12:30:38.090 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 61 67 5f 74 69 64 22 3a 32 32 7d 7d 65 b1 69 60 |ag_tid":22}}e.i`| 2021-04-04T12:30:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 ae b7 a2 0c 00 |.....| 2021-04-04T12:30:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000125 2021-04-04T12:30:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_87727dd3-4985-41eb-bd8e-3591e277f847 2021-04-04T12:30:38.091 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T12:30:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T12:30:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 43 72 5e 0e 00 | replay..i`Cr^..| 2021-04-04T12:30:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.093 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_a744e3fd-757f-4a7c-b80e-63f68a69a413 2021-04-04T12:30:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T12:30:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T12:30:38.094 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 27 b6 7c 0e 00 | replay..i`'.|..| 2021-04-04T12:30:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:38.095 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_a9fc372c-84b2-41cd-a172-f5e0c63e6d90 2021-04-04T12:30:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T12:30:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T12:30:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T12:30:38.096 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:38.097 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T12:30:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T12:30:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T12:30:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T12:30:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T12:30:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T12:30:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 32 7d 2c 22 70 72 69 6d 61 72 |_tid":2},"primar| 2021-04-04T12:30:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T12:30:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T12:30:38.099 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T12:30:38.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 32 7d 7d 41 8a 69 60 82 35 |g_tid":2}}A.i`.5| 2021-04-04T12:30:38.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 32 0b 00 |2..| 2021-04-04T12:30:38.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T12:30:38.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_caa231b2-f85a-456d-b241-95227b74117e 2021-04-04T12:30:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T12:30:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T12:30:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T12:30:38.101 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T12:30:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T12:30:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T12:30:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T12:30:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T12:30:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T12:30:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 31 7d 2c 22 70 72 69 6d 61 72 |_tid":1},"primar| 2021-04-04T12:30:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T12:30:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T12:30:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T12:30:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 31 7d 7d 65 b1 69 60 f4 bb |g_tid":1}}e.i`..| 2021-04-04T12:30:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 a3 0c 00 |...| 2021-04-04T12:30:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T12:30:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_d9317e9f-8ea9-4380-a618-30a1fe0a728a 2021-04-04T12:30:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T12:30:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T12:30:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 14 85 92 0e 00 | replay..i`.....| 2021-04-04T12:30:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.107 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T12:30:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T12:30:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T12:30:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T12:30:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T12:30:38.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T12:30:38.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T12:30:38.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T12:30:38.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T12:30:38.110 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T12:30:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 32 7d 2c 22 70 72 69 6d 61 72 |_tid":2},"primar| 2021-04-04T12:30:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T12:30:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T12:30:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T12:30:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 32 7d 7d 65 b1 69 60 18 6d |g_tid":2}}e.i`.m| 2021-04-04T12:30:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 a4 0c 00 |...| 2021-04-04T12:30:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T12:30:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_ead75b84-b901-49af-ba5f-d9a8188a67ef 2021-04-04T12:30:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T12:30:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T12:30:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 92 89 69 60 20 4e 2e 0c 00 | replay..i` N...| 2021-04-04T12:30:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror ns1 images 2021-04-04T12:30:38.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror ns1 images 2021-04-04T12:30:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 ls -l 2021-04-04T12:30:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:NAME SIZE PARENT FMT PROT LOCK 2021-04-04T12:30:38.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool 128 MiB 2 excl 2021-04-04T12:30:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool@.rbd-mirror.4ecbcd27-d351-4450-9326-27a96f352493.2fc90865-f9af-418c-899c-52a5812d98c1 128 MiB 2 2021-04-04T12:30:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirrorns1 mirror pool info 2021-04-04T12:30:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirrorns1 mirror pool info 2021-04-04T12:30:38.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool info 2021-04-04T12:30:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: pool 2021-04-04T12:30:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirrorns1 mirror pool status 2021-04-04T12:30:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirrorns1 mirror pool status 2021-04-04T12:30:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool status --verbose 2021-04-04T12:30:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: OK 2021-04-04T12:30:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: OK 2021-04-04T12:30:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: OK 2021-04-04T12:30:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 1 total 2021-04-04T12:30:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 1 replaying 2021-04-04T12:30:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2021-04-04T12:30:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4934: 2021-04-04T12:30:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4944 2021-04-04T12:30:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.0 2021-04-04T12:30:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T12:30:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T12:30:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: true 2021-04-04T12:30:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T12:30:38.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4938: 2021-04-04T12:30:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4948 2021-04-04T12:30:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.1 2021-04-04T12:30:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T12:30:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T12:30:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T12:30:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T12:30:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4945: 2021-04-04T12:30:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4957 2021-04-04T12:30:38.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.2 2021-04-04T12:30:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T12:30:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T12:30:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T12:30:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T12:30:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4952: 2021-04-04T12:30:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4961 2021-04-04T12:30:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.3 2021-04-04T12:30:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T12:30:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T12:30:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T12:30:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T12:30:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2021-04-04T12:30:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool: 2021-04-04T12:30:38.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 41466285-b62b-4d8b-b1e4-3b04bb3f224c 2021-04-04T12:30:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":104,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":2}} 2021-04-04T12:30:38.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.1 on smithi122 2021-04-04T12:30:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T12:30:38.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T12:30:38.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:38.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:38.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 ls 2021-04-04T12:30:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool info 2021-04-04T12:30:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool info 2021-04-04T12:30:38.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 info test_data_pool 2021-04-04T12:30:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_data_pool': 2021-04-04T12:30:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2021-04-04T12:30:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1354875c93da 2021-04-04T12:30:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: data_pool: mirror_parent 2021-04-04T12:30:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.3.1354875c93da 2021-04-04T12:30:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling, data-pool 2021-04-04T12:30:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:41:01 2021 2021-04-04T12:30:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:41:01 2021 2021-04-04T12:30:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:41:01 2021 2021-04-04T12:30:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 1354875c93da 2021-04-04T12:30:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 41466285-b62b-4d8b-b1e4-3b04bb3f224c 2021-04-04T12:30:38.336 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T12:30:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool journal status 2021-04-04T12:30:38.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool journal status 2021-04-04T12:30:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 journal status --image test_data_pool 2021-04-04T12:30:38.414 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:38.415 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool snapshots 2021-04-04T12:30:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool snapshots 2021-04-04T12:30:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 snap ls --all test_data_pool 2021-04-04T12:30:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T12:30:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 16 .rbd-mirror.4ecbcd27-d351-4450-9326-27a96f352493.2fc90865-f9af-418c-899c-52a5812d98c1 128 MiB Sun Apr 4 09:41:01 2021 user 2021-04-04T12:30:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror ns1 rbd_mirroring omap vals 2021-04-04T12:30:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror ns1 rbd_mirroring omap vals 2021-04-04T12:30:38.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirroring 2021-04-04T12:30:38.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_41466285-b62b-4d8b-b1e4-3b04bb3f224c 2021-04-04T12:30:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:38.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 33 35 34 38 37 35 63 39 33 64 61 |....1354875c93da| 2021-04-04T12:30:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_1354875c93da 2021-04-04T12:30:38.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 31 34 36 36 32 |..*...$...414662| 2021-04-04T12:30:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 35 2d 62 36 32 62 2d 34 64 38 62 2d 62 31 65 |85-b62b-4d8b-b1e| 2021-04-04T12:30:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 34 2d 33 62 30 34 62 62 33 66 32 32 34 63 01 00 |4-3b04bb3f224c..| 2021-04-04T12:30:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2021-04-04T12:30:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2021-04-04T12:30:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 00 00 00 |....| 2021-04-04T12:30:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2021-04-04T12:30:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2021-04-04T12:30:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2021-04-04T12:30:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 35 35 30 63 35 61 65 62 2d 66 31 64 39 2d 34 38 |550c5aeb-f1d9-48| 2021-04-04T12:30:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 32 31 2d 61 39 31 35 2d 34 34 30 37 33 30 31 63 |21-a915-4407301c| 2021-04-04T12:30:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 65 38 31 34 |e814| 2021-04-04T12:30:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2021-04-04T12:30:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_41466285-b62b-4d8b-b1e4-3b04bb3f224c 2021-04-04T12:30:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (252 bytes) : 2021-04-04T12:30:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 54 13 00 00 00 00 00 00 01 |..,....T........| 2021-04-04T12:30:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 c9 c6 c6 88 10 00 |................| 2021-04-04T12:30:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 c4 00 00 00 04 b6 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 31 30 34 2c 22 65 6e |primary":104,"en| 2021-04-04T12:30:38.538 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 74 72 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 |tries_per_second| 2021-04-04T12:30:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 22 3a 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 |":0.0,"non_prima| 2021-04-04T12:30:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 72 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 7d 2c |ry_position":{},| 2021-04-04T12:30:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 70 72 69 6d 61 72 79 5f 70 6f 73 69 74 69 6f |"primary_positio| 2021-04-04T12:30:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 6e 22 3a 7b 22 65 6e 74 72 79 5f 74 69 64 22 3a |n":{"entry_tid":| 2021-04-04T12:30:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 30 31 2c 22 6f 62 6a 65 63 74 5f 6e 75 6d 62 |101,"object_numb| 2021-04-04T12:30:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 65 72 22 3a 31 2c 22 74 61 67 5f 74 69 64 22 3a |er":1,"tag_tid":| 2021-04-04T12:30:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 32 7d 7d 23 8a 69 60 51 7a fc 0c 00 |2}}#.i`Qz...| 2021-04-04T12:30:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000fc 2021-04-04T12:30:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_909bceaa-c260-415a-a60e-eee31d4f3e89 2021-04-04T12:30:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T12:30:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T12:30:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 ab 89 69 60 28 2d 6f 0c 00 | replay..i`(-o..| 2021-04-04T12:30:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror ns2 images 2021-04-04T12:30:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror ns2 images 2021-04-04T12:30:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 ls -l 2021-04-04T12:30:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirrorns2 mirror pool info 2021-04-04T12:30:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:38.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirrorns2 mirror pool info 2021-04-04T12:30:38.576 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool info 2021-04-04T12:30:38.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: image 2021-04-04T12:30:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirrorns2 mirror pool status 2021-04-04T12:30:38.614 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirrorns2 mirror pool status 2021-04-04T12:30:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool status --verbose 2021-04-04T12:30:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: OK 2021-04-04T12:30:38.658 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: OK 2021-04-04T12:30:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: OK 2021-04-04T12:30:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 0 total 2021-04-04T12:30:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2021-04-04T12:30:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4934: 2021-04-04T12:30:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4944 2021-04-04T12:30:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.0 2021-04-04T12:30:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T12:30:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T12:30:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: true 2021-04-04T12:30:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T12:30:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.661 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4938: 2021-04-04T12:30:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4948 2021-04-04T12:30:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.1 2021-04-04T12:30:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T12:30:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T12:30:38.662 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T12:30:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T12:30:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4945: 2021-04-04T12:30:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4957 2021-04-04T12:30:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.2 2021-04-04T12:30:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T12:30:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T12:30:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T12:30:38.664 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T12:30:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4952: 2021-04-04T12:30:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4961 2021-04-04T12:30:38.665 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.3 2021-04-04T12:30:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T12:30:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T12:30:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T12:30:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T12:30:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2021-04-04T12:30:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 ls 2021-04-04T12:30:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror ns2 rbd_mirroring omap vals 2021-04-04T12:30:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror ns2 rbd_mirroring omap vals 2021-04-04T12:30:38.711 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirroring 2021-04-04T12:30:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2021-04-04T12:30:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2021-04-04T12:30:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 00 00 00 |....| 2021-04-04T12:30:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2021-04-04T12:30:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2021-04-04T12:30:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2021-04-04T12:30:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 30 32 33 35 62 38 37 31 2d 36 38 63 39 2d 34 65 |0235b871-68c9-4e| 2021-04-04T12:30:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 62 61 2d 39 30 62 62 2d 62 34 65 38 36 64 30 32 |ba-90bb-b4e86d02| 2021-04-04T12:30:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 38 35 31 |2851| 2021-04-04T12:30:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2021-04-04T12:30:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_af2694d6-07b0-4e37-95e1-92663262de9a 2021-04-04T12:30:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 50 13 00 00 00 00 00 00 01 |..,....P........| 2021-04-04T12:30:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 2f 9c 2e fc 10 00 |........../.....| 2021-04-04T12:30:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 ab 89 69 60 19 33 7b 0c 00 | replay..i`.3{..| 2021-04-04T12:30:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent images 2021-04-04T12:30:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:38.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parent images 2021-04-04T12:30:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ls -l 2021-04-04T12:30:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:NAME SIZE PARENT FMT PROT LOCK 2021-04-04T12:30:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_parent 128 MiB 2 excl 2021-04-04T12:30:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_parent@snap 128 MiB 2 yes 2021-04-04T12:30:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent mirror pool info 2021-04-04T12:30:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:38.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parent mirror pool info 2021-04-04T12:30:38.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace mirror pool info 2021-04-04T12:30:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: image 2021-04-04T12:30:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Site Name: cluster1 2021-04-04T12:30:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Peer Sites: 2021-04-04T12:30:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:UUID: 062cbc91-4713-4ded-b744-6fda24ecf93a 2021-04-04T12:30:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Name: cluster2 2021-04-04T12:30:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirror UUID: de536629-eb1e-4d40-bad3-7bb7b2b463fd 2021-04-04T12:30:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Direction: rx-tx 2021-04-04T12:30:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Client: client.mirror 2021-04-04T12:30:38.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:38.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent mirror pool status 2021-04-04T12:30:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parent mirror pool status 2021-04-04T12:30:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror_parent --namespace mirror pool status --verbose 2021-04-04T12:30:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: WARNING 2021-04-04T12:30:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: OK 2021-04-04T12:30:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: WARNING 2021-04-04T12:30:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 1 total 2021-04-04T12:30:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 1 unknown 2021-04-04T12:30:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2021-04-04T12:30:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4934: 2021-04-04T12:30:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4964 2021-04-04T12:30:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.0 2021-04-04T12:30:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T12:30:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T12:30:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: true 2021-04-04T12:30:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T12:30:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4938: 2021-04-04T12:30:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 2021-04-04T12:30:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.1 2021-04-04T12:30:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T12:30:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T12:30:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T12:30:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T12:30:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4945: 2021-04-04T12:30:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4975 2021-04-04T12:30:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.2 2021-04-04T12:30:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T12:30:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T12:30:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T12:30:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T12:30:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:service 4952: 2021-04-04T12:30:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: instance_id: 4971 2021-04-04T12:30:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client_id: mirror.3 2021-04-04T12:30:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: hostname: smithi122 2021-04-04T12:30:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: version: 16.2.0-22-gf4fb4141 2021-04-04T12:30:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: leader: false 2021-04-04T12:30:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: OK 2021-04-04T12:30:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.915 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2021-04-04T12:30:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_parent: 2021-04-04T12:30:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: ed3379f2-3a26-4e23-8dd8-e2567a8361ce 2021-04-04T12:30:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":8},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":8}} 2021-04-04T12:30:38.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: service: mirror.0 on smithi122 2021-04-04T12:30:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 12:30:29 2021-04-04T12:30:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:38.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster2 2021-04-04T12:30:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:38.949 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:38.950 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ls 2021-04-04T12:30:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_parent info 2021-04-04T12:30:38.995 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_parent info 2021-04-04T12:30:38.996 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace info test_parent 2021-04-04T12:30:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_parent': 2021-04-04T12:30:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2021-04-04T12:30:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10ed3aaf7c64 2021-04-04T12:30:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10ed3aaf7c64 2021-04-04T12:30:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:33:58 2021 2021-04-04T12:30:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:33:58 2021 2021-04-04T12:30:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:33:58 2021 2021-04-04T12:30:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10ed3aaf7c64 2021-04-04T12:30:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: ed3379f2-3a26-4e23-8dd8-e2567a8361ce 2021-04-04T12:30:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T12:30:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_parent journal status 2021-04-04T12:30:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_parent journal status 2021-04-04T12:30:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace journal status --image test_parent 2021-04-04T12:30:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:39.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:39.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=de536629-eb1e-4d40-bad3-7bb7b2b463fd, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:39.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_parent snapshots 2021-04-04T12:30:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_parent snapshots 2021-04-04T12:30:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace snap ls --all test_parent 2021-04-04T12:30:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T12:30:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 4 snap 128 MiB yes Sun Apr 4 09:33:59 2021 user 2021-04-04T12:30:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent rbd_mirroring omap vals 2021-04-04T12:30:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parent rbd_mirroring omap vals 2021-04-04T12:30:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace listomapvals rbd_mirroring 2021-04-04T12:30:39.259 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_ed3379f2-3a26-4e23-8dd8-e2567a8361ce 2021-04-04T12:30:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 64 33 61 61 66 37 63 36 34 |....10ed3aaf7c64| 2021-04-04T12:30:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10ed3aaf7c64 2021-04-04T12:30:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 64 33 33 37 39 |..*...$...ed3379| 2021-04-04T12:30:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 66 32 2d 33 61 32 36 2d 34 65 32 33 2d 38 64 64 |f2-3a26-4e23-8dd| 2021-04-04T12:30:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 38 2d 65 32 35 36 37 61 38 33 36 31 63 65 01 00 |8-e2567a8361ce..| 2021-04-04T12:30:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2021-04-04T12:30:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2021-04-04T12:30:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 00 00 00 |....| 2021-04-04T12:30:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2021-04-04T12:30:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_peer_062cbc91-4713-4ded-b744-6fda24ecf93a 2021-04-04T12:30:39.266 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (132 bytes) : 2021-04-04T12:30:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 30 36 32 63 62 63 |..~...$...062cbc| 2021-04-04T12:30:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 39 31 2d 34 37 31 33 2d 34 64 65 64 2d 62 37 34 |91-4713-4ded-b74| 2021-04-04T12:30:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 34 2d 36 66 64 61 32 34 65 63 66 39 33 61 08 00 |4-6fda24ecf93a..| 2021-04-04T12:30:39.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2021-04-04T12:30:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2021-04-04T12:30:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 ff ff ff 02 24 00 00 00 64 65 35 33 36 36 32 39 |....$...de536629| 2021-04-04T12:30:39.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2d 65 62 31 65 2d 34 64 34 30 2d 62 61 64 33 2d |-eb1e-4d40-bad3-| 2021-04-04T12:30:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 37 62 62 37 62 32 62 34 36 33 66 64 6e 89 69 60 |7bb7b2b463fdn.i`| 2021-04-04T12:30:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 ab cd 1a 02 |....| 2021-04-04T12:30:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000084 2021-04-04T12:30:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2021-04-04T12:30:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2021-04-04T12:30:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 31 63 64 32 31 66 65 33 2d 62 65 33 62 2d 34 61 |1cd21fe3-be3b-4a| 2021-04-04T12:30:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 35 38 2d 61 63 32 64 2d 61 32 35 64 65 33 31 38 |58-ac2d-a25de318| 2021-04-04T12:30:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 64 65 62 63 |debc| 2021-04-04T12:30:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2021-04-04T12:30:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:status_global_ed3379f2-3a26-4e23-8dd8-e2567a8361ce 2021-04-04T12:30:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T12:30:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 64 13 00 00 00 00 00 00 01 |..,....d........| 2021-04-04T12:30:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 d4 ba e4 e5 10 00 |................| 2021-04-04T12:30:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T12:30:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T12:30:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T12:30:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T12:30:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 31 2c 22 6f 62 6a 65 63 |ry_tid":1,"objec| 2021-04-04T12:30:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 61 67 |t_number":1,"tag| 2021-04-04T12:30:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 38 7d 2c 22 70 72 69 6d 61 72 |_tid":8},"primar| 2021-04-04T12:30:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T12:30:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 31 2c 22 6f 62 6a 65 |try_tid":1,"obje| 2021-04-04T12:30:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 61 |ct_number":1,"ta| 2021-04-04T12:30:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 38 7d 7d 65 b1 69 60 fd 21 |g_tid":8}}e.i`.!| 2021-04-04T12:30:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 e0 12 00 |...| 2021-04-04T12:30:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T12:30:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent ns1 images 2021-04-04T12:30:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parent ns1 images 2021-04-04T12:30:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls -l 2021-04-04T12:30:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns1' does not exist. 2021-04-04T12:30:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: listing images failed: (2) No such file or directory 2021-04-04T12:30:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parentns1 mirror pool info 2021-04-04T12:30:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parentns1 mirror pool info 2021-04-04T12:30:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool info 2021-04-04T12:30:39.343 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns1' does not exist. 2021-04-04T12:30:39.345 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parentns1 mirror pool status 2021-04-04T12:30:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parentns1 mirror pool status 2021-04-04T12:30:39.347 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool status --verbose 2021-04-04T12:30:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns1' does not exist. 2021-04-04T12:30:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls 2021-04-04T12:30:39.428 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent ns1 rbd_mirroring omap vals 2021-04-04T12:30:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parent ns1 rbd_mirroring omap vals 2021-04-04T12:30:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2021-04-04T12:30:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2021-04-04T12:30:39.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent ns2 images 2021-04-04T12:30:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:39.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parent ns2 images 2021-04-04T12:30:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls -l 2021-04-04T12:30:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns2' does not exist. 2021-04-04T12:30:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: listing images failed: (2) No such file or directory 2021-04-04T12:30:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parentns2 mirror pool info 2021-04-04T12:30:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parentns2 mirror pool info 2021-04-04T12:30:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool info 2021-04-04T12:30:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns2' does not exist. 2021-04-04T12:30:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parentns2 mirror pool status 2021-04-04T12:30:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parentns2 mirror pool status 2021-04-04T12:30:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool status --verbose 2021-04-04T12:30:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns2' does not exist. 2021-04-04T12:30:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls 2021-04-04T12:30:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 mirror_parent ns2 rbd_mirroring omap vals 2021-04-04T12:30:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 mirror_parent ns2 rbd_mirroring omap vals 2021-04-04T12:30:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2021-04-04T12:30:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2021-04-04T12:30:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 status 2021-04-04T12:30:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 status 2021-04-04T12:30:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster2 -s 2021-04-04T12:30:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: cluster: 2021-04-04T12:30:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 0fed4950-22e0-476b-9583-b4115f5b795c 2021-04-04T12:30:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: health: HEALTH_OK 2021-04-04T12:30:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: services: 2021-04-04T12:30:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mon: 1 daemons, quorum a (age 3h) 2021-04-04T12:30:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mgr: x(active, since 3h) 2021-04-04T12:30:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: osd: 3 osds: 3 up (since 3h), 3 in (since 3h) 2021-04-04T12:30:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: data: 2021-04-04T12:30:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: pools: 4 pools, 137 pgs 2021-04-04T12:30:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: objects: 502 objects, 1.0 GiB 2021-04-04T12:30:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: usage: 1.4 GiB used, 267 GiB / 268 GiB avail 2021-04-04T12:30:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: pgs: 137 active+clean 2021-04-04T12:30:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: io: 2021-04-04T12:30:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: client: 17 KiB/s rd, 1.1 KiB/s wr, 21 op/s rd, 1 op/s wr 2021-04-04T12:30:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster2 service dump 2021-04-04T12:30:40.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2021-04-04T12:30:40.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "epoch": 65, 2021-04-04T12:30:40.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "modified": "2021-04-04T12:29:36.090139+0000", 2021-04-04T12:30:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "services": {} 2021-04-04T12:30:40.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2021-04-04T12:30:40.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= ceph --cluster cluster2 service status 2021-04-04T12:30:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{} 2021-04-04T12:30:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror images 2021-04-04T12:30:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror images 2021-04-04T12:30:40.957 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ls -l 2021-04-04T12:30:41.039 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:NAME SIZE PARENT FMT PROT LOCK 2021-04-04T12:30:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test 128 MiB 2 2021-04-04T12:30:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test1 128 MiB 2 2021-04-04T12:30:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2 128 MiB 2 2021-04-04T12:30:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2@snap1 128 MiB 2 2021-04-04T12:30:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2@snap2 128 MiB 2 2021-04-04T12:30:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2@snap_rename_20 128 MiB 2 2021-04-04T12:30:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test4 128 MiB 2 2021-04-04T12:30:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test4@snap1 128 MiB 2 yes 2021-04-04T12:30:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test4@snap2 128 MiB 2 yes 2021-04-04T12:30:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone 128 MiB mirror_parent/test_parent@snap 2 2021-04-04T12:30:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone1 128 MiB mirror_parent/test_parent@snap 2 2021-04-04T12:30:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone_v1 128 MiB mirror_parent/test_parent@snap 2 2021-04-04T12:30:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool 128 MiB 2 2021-04-04T12:30:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool@snap1 128 MiB 2 2021-04-04T12:30:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool@snap2 128 MiB 2 2021-04-04T12:30:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_force_promote 128 MiB 2 2021-04-04T12:30:41.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror mirror pool info 2021-04-04T12:30:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror mirror pool info 2021-04-04T12:30:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace mirror pool info 2021-04-04T12:30:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: pool 2021-04-04T12:30:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Site Name: cluster2 2021-04-04T12:30:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Peer Sites: 2021-04-04T12:30:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:UUID: ba5c4a7e-49df-4b8d-99f2-b5377df755f6 2021-04-04T12:30:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Name: cluster1 2021-04-04T12:30:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirror UUID: 4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Direction: rx-tx 2021-04-04T12:30:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Client: client.mirror 2021-04-04T12:30:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror mirror pool status 2021-04-04T12:30:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror mirror pool status 2021-04-04T12:30:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror --namespace mirror pool status --verbose 2021-04-04T12:30:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: UNKNOWN 2021-04-04T12:30:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: UNKNOWN 2021-04-04T12:30:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: OK 2021-04-04T12:30:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 9 total 2021-04-04T12:30:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 7 replaying 2021-04-04T12:30:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2 stopped 2021-04-04T12:30:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2021-04-04T12:30:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: none 2021-04-04T12:30:41.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2021-04-04T12:30:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test: 2021-04-04T12:30:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 3c8112b2-6f20-4a55-833f-e3a0d1e92beb 2021-04-04T12:30:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:41.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T12:30:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:41.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T12:30:41.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 12:30:29 2021-04-04T12:30:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test1: 2021-04-04T12:30:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T12:30:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:41.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T12:30:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T12:30:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 12:30:29 2021-04-04T12:30:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test2: 2021-04-04T12:30:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 80ed7469-0640-4a03-9952-a34e4d433832 2021-04-04T12:30:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T12:30:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":22},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":22}} 2021-04-04T12:30:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 12:30:29 2021-04-04T12:30:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test4: 2021-04-04T12:30:41.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 698bca8d-5596-41e0-b6a2-71fc39245e79 2021-04-04T12:30:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:41.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T12:30:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T12:30:41.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:43:29 2021-04-04T12:30:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone: 2021-04-04T12:30:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: a9fc372c-84b2-41cd-a172-f5e0c63e6d90 2021-04-04T12:30:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T12:30:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":2}} 2021-04-04T12:30:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:43:29 2021-04-04T12:30:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone1: 2021-04-04T12:30:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 14c06da4-7ebf-41e7-8aa7-0b0a1328684a 2021-04-04T12:30:41.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:41.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:41.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T12:30:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T12:30:41.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: local image is primary 2021-04-04T12:30:41.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:43:24 2021-04-04T12:30:41.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_clone_v1: 2021-04-04T12:30:41.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: caa231b2-f85a-456d-b241-95227b74117e 2021-04-04T12:30:41.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:41.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T12:30:41.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:41.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1},"primary_position":{"entry_tid":3,"object_number":3,"tag_tid":1}} 2021-04-04T12:30:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 12:30:29 2021-04-04T12:30:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool: 2021-04-04T12:30:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 63d2ca14-0fae-4993-96ee-cffbec8f3ffc 2021-04-04T12:30:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:41.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T12:30:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":5}} 2021-04-04T12:30:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:43:29 2021-04-04T12:30:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_force_promote: 2021-04-04T12:30:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T12:30:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T12:30:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+stopped 2021-04-04T12:30:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: local image is primary 2021-04-04T12:30:41.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 12:30:39 2021-04-04T12:30:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ls 2021-04-04T12:30:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test info 2021-04-04T12:30:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test info 2021-04-04T12:30:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test 2021-04-04T12:30:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2021-04-04T12:30:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T12:30:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1099ce7c2fd0 2021-04-04T12:30:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.1099ce7c2fd0 2021-04-04T12:30:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:30:50 2021 2021-04-04T12:30:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:40:44 2021 2021-04-04T12:30:41.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:32:37 2021 2021-04-04T12:30:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 1099ce7c2fd0 2021-04-04T12:30:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:41.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 3c8112b2-6f20-4a55-833f-e3a0d1e92beb 2021-04-04T12:30:41.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T12:30:41.277 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test journal status 2021-04-04T12:30:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test journal status 2021-04-04T12:30:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test 2021-04-04T12:30:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T12:30:41.335 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T12:30:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test snapshots 2021-04-04T12:30:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test snapshots 2021-04-04T12:30:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test 2021-04-04T12:30:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test1 info 2021-04-04T12:30:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test1 info 2021-04-04T12:30:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test1 2021-04-04T12:30:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test1': 2021-04-04T12:30:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T12:30:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 10e1f6d670b3 2021-04-04T12:30:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.10e1f6d670b3 2021-04-04T12:30:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:31:05 2021 2021-04-04T12:30:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:31:05 2021 2021-04-04T12:30:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:31:05 2021 2021-04-04T12:30:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 10e1f6d670b3 2021-04-04T12:30:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T12:30:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T12:30:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test1 journal status 2021-04-04T12:30:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test1 journal status 2021-04-04T12:30:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test1 2021-04-04T12:30:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:41.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T12:30:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T12:30:41.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.519 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test1 snapshots 2021-04-04T12:30:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test1 snapshots 2021-04-04T12:30:41.520 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test1 2021-04-04T12:30:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test2 info 2021-04-04T12:30:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test2 info 2021-04-04T12:30:41.573 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test2 2021-04-04T12:30:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test2': 2021-04-04T12:30:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:41.628 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 3 2021-04-04T12:30:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12c761e38e1d 2021-04-04T12:30:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12c761e38e1d 2021-04-04T12:30:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:50 2021 2021-04-04T12:30:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:50 2021 2021-04-04T12:30:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:50 2021 2021-04-04T12:30:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 12c761e38e1d 2021-04-04T12:30:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 80ed7469-0640-4a03-9952-a34e4d433832 2021-04-04T12:30:41.632 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T12:30:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.633 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test2 journal status 2021-04-04T12:30:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test2 journal status 2021-04-04T12:30:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test2 2021-04-04T12:30:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:41.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:41.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:41.697 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=1, tag_tid=22, entry_tid=1], [object_number=0, tag_tid=22, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2]]], state=connected] 2021-04-04T12:30:41.698 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=22, entry_tid=1], [object_number=0, tag_tid=22, entry_tid=0], [object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2]]], state=connected] 2021-04-04T12:30:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test2 snapshots 2021-04-04T12:30:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test2 snapshots 2021-04-04T12:30:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test2 2021-04-04T12:30:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T12:30:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 20 snap1 128 MiB Sun Apr 4 09:34:51 2021 user 2021-04-04T12:30:41.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 21 snap2 128 MiB Sun Apr 4 09:34:52 2021 user 2021-04-04T12:30:41.758 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 44 snap_rename_20 128 MiB Sun Apr 4 09:39:55 2021 user 2021-04-04T12:30:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.763 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test4 info 2021-04-04T12:30:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test4 info 2021-04-04T12:30:41.764 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test4 2021-04-04T12:30:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test4': 2021-04-04T12:30:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2021-04-04T12:30:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 12e57718fb70 2021-04-04T12:30:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.12e57718fb70 2021-04-04T12:30:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:35:01 2021 2021-04-04T12:30:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:35:01 2021 2021-04-04T12:30:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:35:01 2021 2021-04-04T12:30:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 12e57718fb70 2021-04-04T12:30:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 698bca8d-5596-41e0-b6a2-71fc39245e79 2021-04-04T12:30:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T12:30:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test4 journal status 2021-04-04T12:30:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test4 journal status 2021-04-04T12:30:41.835 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test4 2021-04-04T12:30:41.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T12:30:41.904 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T12:30:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test4 snapshots 2021-04-04T12:30:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test4 snapshots 2021-04-04T12:30:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test4 2021-04-04T12:30:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T12:30:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 28 snap1 128 MiB yes Sun Apr 4 09:35:02 2021 user 2021-04-04T12:30:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 29 snap2 128 MiB yes Sun Apr 4 09:35:03 2021 user 2021-04-04T12:30:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone info 2021-04-04T12:30:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone info 2021-04-04T12:30:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test_clone 2021-04-04T12:30:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_clone': 2021-04-04T12:30:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T12:30:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1226aee074ef 2021-04-04T12:30:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.1226aee074ef 2021-04-04T12:30:42.046 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:42.047 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:42.048 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:33:56 2021 2021-04-04T12:30:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:33:56 2021 2021-04-04T12:30:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:33:56 2021 2021-04-04T12:30:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: parent: mirror_parent/test_parent@snap 2021-04-04T12:30:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: overlap: 128 MiB 2021-04-04T12:30:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 1226aee074ef 2021-04-04T12:30:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: a9fc372c-84b2-41cd-a172-f5e0c63e6d90 2021-04-04T12:30:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T12:30:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:42.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:42.055 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone journal status 2021-04-04T12:30:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone journal status 2021-04-04T12:30:42.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test_clone 2021-04-04T12:30:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:42.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T12:30:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T12:30:42.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone snapshots 2021-04-04T12:30:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone snapshots 2021-04-04T12:30:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test_clone 2021-04-04T12:30:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone1 info 2021-04-04T12:30:42.210 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone1 info 2021-04-04T12:30:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test_clone1 2021-04-04T12:30:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_clone1': 2021-04-04T12:30:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T12:30:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 113b92c6dbf7 2021-04-04T12:30:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.113b92c6dbf7 2021-04-04T12:30:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:42.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:14 2021 2021-04-04T12:30:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:14 2021 2021-04-04T12:30:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:14 2021 2021-04-04T12:30:42.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: parent: mirror_parent/test_parent@snap 2021-04-04T12:30:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: overlap: 128 MiB 2021-04-04T12:30:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 113b92c6dbf7 2021-04-04T12:30:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 14c06da4-7ebf-41e7-8aa7-0b0a1328684a 2021-04-04T12:30:42.296 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: false 2021-04-04T12:30:42.297 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:42.298 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:42.299 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone1 journal status 2021-04-04T12:30:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone1 journal status 2021-04-04T12:30:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test_clone1 2021-04-04T12:30:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone1 snapshots 2021-04-04T12:30:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone1 snapshots 2021-04-04T12:30:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test_clone1 2021-04-04T12:30:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:42.454 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone_v1 info 2021-04-04T12:30:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone_v1 info 2021-04-04T12:30:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test_clone_v1 2021-04-04T12:30:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_clone_v1': 2021-04-04T12:30:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:42.522 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T12:30:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 125dca854283 2021-04-04T12:30:42.523 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.125dca854283 2021-04-04T12:30:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:42.524 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:42.525 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:12 2021 2021-04-04T12:30:42.526 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:12 2021 2021-04-04T12:30:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:12 2021 2021-04-04T12:30:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: parent: mirror_parent/test_parent@snap 2021-04-04T12:30:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: overlap: 128 MiB 2021-04-04T12:30:42.528 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 125dca854283 2021-04-04T12:30:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:42.529 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: caa231b2-f85a-456d-b241-95227b74117e 2021-04-04T12:30:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T12:30:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone_v1 journal status 2021-04-04T12:30:42.533 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone_v1 journal status 2021-04-04T12:30:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test_clone_v1 2021-04-04T12:30:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T12:30:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=1, entry_tid=3], [object_number=2, tag_tid=1, entry_tid=2], [object_number=1, tag_tid=1, entry_tid=1], [object_number=0, tag_tid=1, entry_tid=0]]], state=connected] 2021-04-04T12:30:42.610 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_clone_v1 snapshots 2021-04-04T12:30:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_clone_v1 snapshots 2021-04-04T12:30:42.612 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test_clone_v1 2021-04-04T12:30:42.679 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool info 2021-04-04T12:30:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool info 2021-04-04T12:30:42.681 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test_data_pool 2021-04-04T12:30:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_data_pool': 2021-04-04T12:30:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 2 2021-04-04T12:30:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1284d6a187ce 2021-04-04T12:30:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: data_pool: mirror_parent 2021-04-04T12:30:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.3.1284d6a187ce 2021-04-04T12:30:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling, data-pool 2021-04-04T12:30:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:34:37 2021 2021-04-04T12:30:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:34:37 2021 2021-04-04T12:30:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:34:37 2021 2021-04-04T12:30:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 1284d6a187ce 2021-04-04T12:30:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 63d2ca14-0fae-4993-96ee-cffbec8f3ffc 2021-04-04T12:30:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T12:30:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool journal status 2021-04-04T12:30:42.749 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool journal status 2021-04-04T12:30:42.750 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test_data_pool 2021-04-04T12:30:42.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T12:30:42.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=1, tag_tid=5, entry_tid=101], [object_number=0, tag_tid=5, entry_tid=100], [object_number=3, tag_tid=5, entry_tid=99], [object_number=2, tag_tid=5, entry_tid=98]]], state=connected] 2021-04-04T12:30:42.819 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool snapshots 2021-04-04T12:30:42.820 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool snapshots 2021-04-04T12:30:42.821 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test_data_pool 2021-04-04T12:30:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T12:30:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 13 snap1 128 MiB Sun Apr 4 09:34:44 2021 user 2021-04-04T12:30:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 14 snap2 128 MiB Sun Apr 4 09:34:45 2021 user 2021-04-04T12:30:42.881 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_force_promote info 2021-04-04T12:30:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_force_promote info 2021-04-04T12:30:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace info test_force_promote 2021-04-04T12:30:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_force_promote': 2021-04-04T12:30:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T12:30:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 11fbaec917d8 2021-04-04T12:30:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.11fbaec917d8 2021-04-04T12:30:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:33:41 2021 2021-04-04T12:30:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:33:41 2021 2021-04-04T12:30:42.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:33:41 2021 2021-04-04T12:30:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 11fbaec917d8 2021-04-04T12:30:42.942 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T12:30:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T12:30:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_force_promote journal status 2021-04-04T12:30:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_force_promote journal status 2021-04-04T12:30:42.947 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace journal status --image test_force_promote 2021-04-04T12:30:43.006 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=1, tag_tid=3, entry_tid=101], [object_number=0, tag_tid=3, entry_tid=100], [object_number=3, tag_tid=3, entry_tid=99], [object_number=2, tag_tid=3, entry_tid=98]]], state=connected] 2021-04-04T12:30:43.007 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[[object_number=3, tag_tid=2, entry_tid=3], [object_number=2, tag_tid=2, entry_tid=2], [object_number=1, tag_tid=2, entry_tid=1], [object_number=0, tag_tid=2, entry_tid=0]]], state=connected] 2021-04-04T12:30:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.010 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_force_promote snapshots 2021-04-04T12:30:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_force_promote snapshots 2021-04-04T12:30:43.012 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace snap ls --all test_force_promote 2021-04-04T12:30:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:43.072 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror rbd_mirroring omap vals 2021-04-04T12:30:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror rbd_mirroring omap vals 2021-04-04T12:30:43.073 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror --namespace listomapvals rbd_mirroring 2021-04-04T12:30:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_14c06da4-7ebf-41e7-8aa7-0b0a1328684a 2021-04-04T12:30:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 31 33 62 39 32 63 36 64 62 66 37 |....113b92c6dbf7| 2021-04-04T12:30:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_3c8112b2-6f20-4a55-833f-e3a0d1e92beb 2021-04-04T12:30:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 39 39 63 65 37 63 32 66 64 30 |....1099ce7c2fd0| 2021-04-04T12:30:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_439e76f7-e9cb-4f92-810d-cf7e95303d79 2021-04-04T12:30:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 31 66 62 61 65 63 39 31 37 64 38 |....11fbaec917d8| 2021-04-04T12:30:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_63d2ca14-0fae-4993-96ee-cffbec8f3ffc 2021-04-04T12:30:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 38 34 64 36 61 31 38 37 63 65 |....1284d6a187ce| 2021-04-04T12:30:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.120 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_698bca8d-5596-41e0-b6a2-71fc39245e79 2021-04-04T12:30:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 65 35 37 37 31 38 66 62 37 30 |....12e57718fb70| 2021-04-04T12:30:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_80ed7469-0640-4a03-9952-a34e4d433832 2021-04-04T12:30:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 63 37 36 31 65 33 38 65 31 64 |....12c761e38e1d| 2021-04-04T12:30:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:43.122 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_a9fc372c-84b2-41cd-a172-f5e0c63e6d90 2021-04-04T12:30:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 32 36 61 65 65 30 37 34 65 66 |....1226aee074ef| 2021-04-04T12:30:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_caa231b2-f85a-456d-b241-95227b74117e 2021-04-04T12:30:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 35 64 63 61 38 35 34 32 38 33 |....125dca854283| 2021-04-04T12:30:43.124 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_e1abe5c5-a773-49e4-9409-b2bf79cfde31 2021-04-04T12:30:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 30 65 31 66 36 64 36 37 30 62 33 |....10e1f6d670b3| 2021-04-04T12:30:43.125 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_1099ce7c2fd0 2021-04-04T12:30:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:43.126 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 33 63 38 31 31 32 |..*...$...3c8112| 2021-04-04T12:30:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 62 32 2d 36 66 32 30 2d 34 61 35 35 2d 38 33 33 |b2-6f20-4a55-833| 2021-04-04T12:30:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 66 2d 65 33 61 30 64 31 65 39 32 62 65 62 01 00 |f-e3a0d1e92beb..| 2021-04-04T12:30:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:43.127 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_10e1f6d670b3 2021-04-04T12:30:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 31 61 62 65 35 |..*...$...e1abe5| 2021-04-04T12:30:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 63 35 2d 61 37 37 33 2d 34 39 65 34 2d 39 34 30 |c5-a773-49e4-940| 2021-04-04T12:30:43.128 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 39 2d 62 32 62 66 37 39 63 66 64 65 33 31 01 00 |9-b2bf79cfde31..| 2021-04-04T12:30:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_113b92c6dbf7 2021-04-04T12:30:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:43.129 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 31 34 63 30 36 64 |..*...$...14c06d| 2021-04-04T12:30:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 61 34 2d 37 65 62 66 2d 34 31 65 37 2d 38 61 61 |a4-7ebf-41e7-8aa| 2021-04-04T12:30:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 37 2d 30 62 30 61 31 33 32 38 36 38 34 61 01 00 |7-0b0a1328684a..| 2021-04-04T12:30:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.130 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_11fbaec917d8 2021-04-04T12:30:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 33 39 65 37 36 |..*...$...439e76| 2021-04-04T12:30:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 66 37 2d 65 39 63 62 2d 34 66 39 32 2d 38 31 30 |f7-e9cb-4f92-810| 2021-04-04T12:30:43.131 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 64 2d 63 66 37 65 39 35 33 30 33 64 37 39 01 00 |d-cf7e95303d79..| 2021-04-04T12:30:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_1226aee074ef 2021-04-04T12:30:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 61 39 66 63 33 37 |..*...$...a9fc37| 2021-04-04T12:30:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 32 63 2d 38 34 62 32 2d 34 31 63 64 2d 61 31 37 |2c-84b2-41cd-a17| 2021-04-04T12:30:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 2d 66 35 65 30 63 36 33 65 36 64 39 30 01 00 |2-f5e0c63e6d90..| 2021-04-04T12:30:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_125dca854283 2021-04-04T12:30:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 63 61 61 32 33 31 |..*...$...caa231| 2021-04-04T12:30:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 62 32 2d 66 38 35 61 2d 34 35 36 64 2d 62 32 34 |b2-f85a-456d-b24| 2021-04-04T12:30:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 31 2d 39 35 32 32 37 62 37 34 31 31 37 65 01 00 |1-95227b74117e..| 2021-04-04T12:30:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_1284d6a187ce 2021-04-04T12:30:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 33 64 32 63 61 |..*...$...63d2ca| 2021-04-04T12:30:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 31 34 2d 30 66 61 65 2d 34 39 39 33 2d 39 36 65 |14-0fae-4993-96e| 2021-04-04T12:30:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 65 2d 63 66 66 62 65 63 38 66 33 66 66 63 01 00 |e-cffbec8f3ffc..| 2021-04-04T12:30:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12c761e38e1d 2021-04-04T12:30:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 38 30 65 64 37 34 |..*...$...80ed74| 2021-04-04T12:30:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 36 39 2d 30 36 34 30 2d 34 61 30 33 2d 39 39 35 |69-0640-4a03-995| 2021-04-04T12:30:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 2d 61 33 34 65 34 64 34 33 33 38 33 32 01 00 |2-a34e4d433832..| 2021-04-04T12:30:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_12e57718fb70 2021-04-04T12:30:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 36 39 38 62 63 61 |..*...$...698bca| 2021-04-04T12:30:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 64 2d 35 35 39 36 2d 34 31 65 30 2d 62 36 61 |8d-5596-41e0-b6a| 2021-04-04T12:30:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 2d 37 31 66 63 33 39 32 34 35 65 37 39 01 00 |2-71fc39245e79..| 2021-04-04T12:30:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2021-04-04T12:30:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2021-04-04T12:30:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 00 00 00 |....| 2021-04-04T12:30:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2021-04-04T12:30:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_peer_ba5c4a7e-49df-4b8d-99f2-b5377df755f6 2021-04-04T12:30:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (132 bytes) : 2021-04-04T12:30:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 62 61 35 63 34 61 |..~...$...ba5c4a| 2021-04-04T12:30:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 37 65 2d 34 39 64 66 2d 34 62 38 64 2d 39 39 66 |7e-49df-4b8d-99f| 2021-04-04T12:30:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 32 2d 62 35 33 37 37 64 66 37 35 35 66 36 08 00 |2-b5377df755f6..| 2021-04-04T12:30:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 63 6c 75 73 74 65 72 31 0d 00 00 00 63 6c |..cluster1....cl| 2021-04-04T12:30:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2021-04-04T12:30:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 ff ff ff 02 24 00 00 00 34 65 63 62 63 64 32 37 |....$...4ecbcd27| 2021-04-04T12:30:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2d 64 33 35 31 2d 34 34 35 30 2d 39 33 32 36 2d |-d351-4450-9326-| 2021-04-04T12:30:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 32 37 61 39 36 66 33 35 32 34 39 33 67 b1 69 60 |27a96f352493g.i`| 2021-04-04T12:30:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 1f ee fd 1d |....| 2021-04-04T12:30:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000084 2021-04-04T12:30:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2021-04-04T12:30:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2021-04-04T12:30:43.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 66 33 36 36 33 35 62 38 2d 34 37 33 37 2d 34 33 |f36635b8-4737-43| 2021-04-04T12:30:43.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 36 35 2d 38 31 36 66 2d 33 64 39 36 39 30 63 65 |65-816f-3d9690ce| 2021-04-04T12:30:43.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 39 35 61 37 |95a7| 2021-04-04T12:30:43.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2021-04-04T12:30:43.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_09a6fbfb-f68f-4e71-98d3-0b0c50ff9d74_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T12:30:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 d4 e6 7c 0d 00 | replay..i`..|..| 2021-04-04T12:30:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_14c06da4-7ebf-41e7-8aa7-0b0a1328684a_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (92 bytes) : 2021-04-04T12:30:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T12:30:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2021-04-04T12:30:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2021-04-04T12:30:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 61 72 79 3c 8a 69 60 18 af ad 0c 00 |ary<.i`.....| 2021-04-04T12:30:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000005c 2021-04-04T12:30:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_19dd5345-25d8-4554-8034-6c31e0df778e_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T12:30:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 0d 32 d0 0d 00 | replay..i`.2...| 2021-04-04T12:30:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_3c8112b2-6f20-4a55-833f-e3a0d1e92beb_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T12:30:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 f1 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 bc 02 0c 40 10 00 |.............@..| 2021-04-04T12:30:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T12:30:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T12:30:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T12:30:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T12:30:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T12:30:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T12:30:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 31 7d 2c 22 70 72 69 6d 61 72 |_tid":1},"primar| 2021-04-04T12:30:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T12:30:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T12:30:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T12:30:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 31 7d 7d 65 b1 69 60 bc 17 |g_tid":1}}e.i`..| 2021-04-04T12:30:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 88 10 00 |...| 2021-04-04T12:30:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T12:30:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_409477f3-cbd2-4eff-9e17-c87e509f4e40_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T12:30:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 90 37 b1 0d 00 | replay..i`.7...| 2021-04-04T12:30:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:43.159 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_439e76f7-e9cb-4f92-810d-cf7e95303d79_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (92 bytes) : 2021-04-04T12:30:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.160 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T12:30:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 24 00 00 00 06 16 00 00 00 6c 6f 63 |....$........loc| 2021-04-04T12:30:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 61 6c 20 69 6d 61 67 65 20 69 73 20 70 72 69 6d |al image is prim| 2021-04-04T12:30:43.161 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 61 72 79 6f b1 69 60 d3 88 b6 1a 00 |aryo.i`.....| 2021-04-04T12:30:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:0000005c 2021-04-04T12:30:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_63d2ca14-0fae-4993-96ee-cffbec8f3ffc_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (295 bytes) : 2021-04-04T12:30:43.162 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T12:30:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 ef 00 00 00 04 e1 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T12:30:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T12:30:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T12:30:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T12:30:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 31 30 31 2c 22 6f 62 6a |ry_tid":101,"obj| 2021-04-04T12:30:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 65 63 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 |ect_number":1,"t| 2021-04-04T12:30:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 61 67 5f 74 69 64 22 3a 35 7d 2c 22 70 72 69 6d |ag_tid":5},"prim| 2021-04-04T12:30:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 61 72 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 |ary_position":{"| 2021-04-04T12:30:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 65 6e 74 72 79 5f 74 69 64 22 3a 31 30 31 2c 22 |entry_tid":101,"| 2021-04-04T12:30:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 6f 62 6a 65 63 74 5f 6e 75 6d 62 65 72 22 3a 31 |object_number":1| 2021-04-04T12:30:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 2c 22 74 61 67 5f 74 69 64 22 3a 35 7d 7d 41 8a |,"tag_tid":5}}A.| 2021-04-04T12:30:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 69 60 9b 13 c5 0b 00 |i`.....| 2021-04-04T12:30:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000127 2021-04-04T12:30:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_690ce498-199d-4023-9c64-ff4d7599e591_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T12:30:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:43.168 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 bc 4a f5 0d 00 | replay..i`.J...| 2021-04-04T12:30:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_698bca8d-5596-41e0-b6a2-71fc39245e79_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.169 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T12:30:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T12:30:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T12:30:43.171 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T12:30:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T12:30:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T12:30:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T12:30:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T12:30:43.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 31 7d 2c 22 70 72 69 6d 61 72 |_tid":1},"primar| 2021-04-04T12:30:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T12:30:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T12:30:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T12:30:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 31 7d 7d 41 8a 69 60 9f 87 |g_tid":1}}A.i`..| 2021-04-04T12:30:43.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 c5 0b 00 |...| 2021-04-04T12:30:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T12:30:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_80ed7469-0640-4a03-9952-a34e4d433832_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (293 bytes) : 2021-04-04T12:30:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T12:30:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 ed 00 00 00 04 df 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T12:30:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T12:30:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T12:30:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T12:30:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 31 2c 22 6f 62 6a 65 63 |ry_tid":1,"objec| 2021-04-04T12:30:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 61 67 |t_number":1,"tag| 2021-04-04T12:30:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 32 32 7d 2c 22 70 72 69 6d 61 |_tid":22},"prima| 2021-04-04T12:30:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 72 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 |ry_position":{"e| 2021-04-04T12:30:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 6e 74 72 79 5f 74 69 64 22 3a 31 2c 22 6f 62 6a |ntry_tid":1,"obj| 2021-04-04T12:30:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 65 63 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 |ect_number":1,"t| 2021-04-04T12:30:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 61 67 5f 74 69 64 22 3a 32 32 7d 7d 65 b1 69 60 |ag_tid":22}}e.i`| 2021-04-04T12:30:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 b2 e5 d4 0c 00 |.....| 2021-04-04T12:30:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000125 2021-04-04T12:30:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_87727dd3-4985-41eb-bd8e-3591e277f847_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T12:30:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 76 8d e2 0d 00 | replay..i`v....| 2021-04-04T12:30:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_a744e3fd-757f-4a7c-b80e-63f68a69a413_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T12:30:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 3e f4 1e 0e 00 | replay..i`>....| 2021-04-04T12:30:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_a9fc372c-84b2-41cd-a172-f5e0c63e6d90_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T12:30:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T12:30:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T12:30:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T12:30:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T12:30:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T12:30:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T12:30:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T12:30:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 32 7d 2c 22 70 72 69 6d 61 72 |_tid":2},"primar| 2021-04-04T12:30:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T12:30:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T12:30:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T12:30:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 32 7d 7d 41 8a 69 60 db d9 |g_tid":2}}A.i`..| 2021-04-04T12:30:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 c5 0b 00 |...| 2021-04-04T12:30:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T12:30:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_caa231b2-f85a-456d-b241-95227b74117e_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.188 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T12:30:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T12:30:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T12:30:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T12:30:43.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T12:30:43.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T12:30:43.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T12:30:43.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T12:30:43.191 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 31 7d 2c 22 70 72 69 6d 61 72 |_tid":1},"primar| 2021-04-04T12:30:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T12:30:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T12:30:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T12:30:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 31 7d 7d 65 b1 69 60 6a ef |g_tid":1}}e.i`j.| 2021-04-04T12:30:43.192 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 d6 0c 00 |...| 2021-04-04T12:30:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T12:30:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_d9317e9f-8ea9-4380-a618-30a1fe0a728a_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T12:30:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 91 89 69 60 e2 e7 34 0e 00 | replay..i`..4..| 2021-04-04T12:30:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_e1abe5c5-a773-49e4-9409-b2bf79cfde31_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T12:30:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T12:30:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:43.196 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T12:30:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T12:30:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T12:30:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T12:30:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 63 |ry_tid":3,"objec| 2021-04-04T12:30:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 67 |t_number":3,"tag| 2021-04-04T12:30:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 32 7d 2c 22 70 72 69 6d 61 72 |_tid":2},"primar| 2021-04-04T12:30:43.198 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T12:30:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 33 2c 22 6f 62 6a 65 |try_tid":3,"obje| 2021-04-04T12:30:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 33 2c 22 74 61 |ct_number":3,"ta| 2021-04-04T12:30:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 32 7d 7d 65 b1 69 60 94 b8 |g_tid":2}}e.i`..| 2021-04-04T12:30:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 d8 0c 00 |...| 2021-04-04T12:30:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T12:30:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_ead75b84-b901-49af-ba5f-d9a8188a67ef_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T12:30:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 92 89 69 60 be 09 2b 0c 00 | replay..i`..+..| 2021-04-04T12:30:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror ns1 images 2021-04-04T12:30:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror ns1 images 2021-04-04T12:30:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 ls -l 2021-04-04T12:30:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:NAME SIZE PARENT FMT PROT LOCK 2021-04-04T12:30:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool 128 MiB 2 2021-04-04T12:30:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirrorns1 mirror pool info 2021-04-04T12:30:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirrorns1 mirror pool info 2021-04-04T12:30:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool info 2021-04-04T12:30:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: pool 2021-04-04T12:30:43.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirrorns1 mirror pool status 2021-04-04T12:30:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirrorns1 mirror pool status 2021-04-04T12:30:43.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool status --verbose 2021-04-04T12:30:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: WARNING 2021-04-04T12:30:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: UNKNOWN 2021-04-04T12:30:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: WARNING 2021-04-04T12:30:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 1 total 2021-04-04T12:30:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 1 unknown 2021-04-04T12:30:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2021-04-04T12:30:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: none 2021-04-04T12:30:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2021-04-04T12:30:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_data_pool: 2021-04-04T12:30:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: 41466285-b62b-4d8b-b1e4-3b04bb3f224c 2021-04-04T12:30:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T12:30:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":104,"entries_per_second":0.0,"non_primary_position":{},"primary_position":{"entry_tid":101,"object_number":1,"tag_tid":2}} 2021-04-04T12:30:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 09:42:59 2021-04-04T12:30:43.293 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 ls 2021-04-04T12:30:43.331 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool info 2021-04-04T12:30:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool info 2021-04-04T12:30:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 info test_data_pool 2021-04-04T12:30:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_data_pool': 2021-04-04T12:30:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T12:30:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1438178462ae 2021-04-04T12:30:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: data_pool: mirror_parent 2021-04-04T12:30:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.3.1438178462ae 2021-04-04T12:30:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling, data-pool 2021-04-04T12:30:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:40:59 2021 2021-04-04T12:30:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:40:59 2021 2021-04-04T12:30:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:40:59 2021 2021-04-04T12:30:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 1438178462ae 2021-04-04T12:30:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: 41466285-b62b-4d8b-b1e4-3b04bb3f224c 2021-04-04T12:30:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T12:30:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool journal status 2021-04-04T12:30:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool journal status 2021-04-04T12:30:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 journal status --image test_data_pool 2021-04-04T12:30:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:43.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=1, tag_tid=2, entry_tid=101], [object_number=0, tag_tid=2, entry_tid=100], [object_number=3, tag_tid=2, entry_tid=99], [object_number=2, tag_tid=2, entry_tid=98]]], state=connected] 2021-04-04T12:30:43.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=4ecbcd27-d351-4450-9326-27a96f352493, commit_position=[positions=[]], state=connected] 2021-04-04T12:30:43.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_data_pool snapshots 2021-04-04T12:30:43.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_data_pool snapshots 2021-04-04T12:30:43.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 snap ls --all test_data_pool 2021-04-04T12:30:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror ns1 rbd_mirroring omap vals 2021-04-04T12:30:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror ns1 rbd_mirroring omap vals 2021-04-04T12:30:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirroring 2021-04-04T12:30:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_41466285-b62b-4d8b-b1e4-3b04bb3f224c 2021-04-04T12:30:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 34 33 38 31 37 38 34 36 32 61 65 |....1438178462ae| 2021-04-04T12:30:43.777 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:43.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_1438178462ae 2021-04-04T12:30:43.778 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:43.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 34 31 34 36 36 32 |..*...$...414662| 2021-04-04T12:30:43.779 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 35 2d 62 36 32 62 2d 34 64 38 62 2d 62 31 65 |85-b62b-4d8b-b1e| 2021-04-04T12:30:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 34 2d 33 62 30 34 62 62 33 66 32 32 34 63 01 00 |4-3b04bb3f224c..| 2021-04-04T12:30:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2021-04-04T12:30:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2021-04-04T12:30:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 00 00 00 |....| 2021-04-04T12:30:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2021-04-04T12:30:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2021-04-04T12:30:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2021-04-04T12:30:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 37 39 36 34 35 34 65 61 2d 33 31 35 62 2d 34 62 |796454ea-315b-4b| 2021-04-04T12:30:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 64 65 2d 38 62 66 33 2d 31 32 36 65 33 34 31 64 |de-8bf3-126e341d| 2021-04-04T12:30:43.785 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 65 32 62 36 |e2b6| 2021-04-04T12:30:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2021-04-04T12:30:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_41466285-b62b-4d8b-b1e4-3b04bb3f224c_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (252 bytes) : 2021-04-04T12:30:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 ee 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 62 52 9a 86 10 00 |..........bR....| 2021-04-04T12:30:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.788 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 c4 00 00 00 04 b6 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:43.789 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 31 30 34 2c 22 65 6e |primary":104,"en| 2021-04-04T12:30:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 74 72 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 |tries_per_second| 2021-04-04T12:30:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 22 3a 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 |":0.0,"non_prima| 2021-04-04T12:30:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 72 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 7d 2c |ry_position":{},| 2021-04-04T12:30:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 22 70 72 69 6d 61 72 79 5f 70 6f 73 69 74 69 6f |"primary_positio| 2021-04-04T12:30:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 6e 22 3a 7b 22 65 6e 74 72 79 5f 74 69 64 22 3a |n":{"entry_tid":| 2021-04-04T12:30:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 31 30 31 2c 22 6f 62 6a 65 63 74 5f 6e 75 6d 62 |101,"object_numb| 2021-04-04T12:30:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 65 72 22 3a 31 2c 22 74 61 67 5f 74 69 64 22 3a |er":1,"tag_tid":| 2021-04-04T12:30:43.793 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 32 7d 7d 23 8a 69 60 b1 09 0c 0d 00 |2}}#.i`.....| 2021-04-04T12:30:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000fc 2021-04-04T12:30:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.794 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_909bceaa-c260-415a-a60e-eee31d4f3e89_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:43.795 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T12:30:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:43.796 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 ab 89 69 60 f0 32 68 0c 00 | replay..i`.2h..| 2021-04-04T12:30:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.798 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.799 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror ns2 images 2021-04-04T12:30:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror ns2 images 2021-04-04T12:30:43.800 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 ls -l 2021-04-04T12:30:43.837 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:NAME SIZE PARENT FMT PROT LOCK 2021-04-04T12:30:43.838 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test 128 MiB 2 2021-04-04T12:30:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirrorns2 mirror pool info 2021-04-04T12:30:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirrorns2 mirror pool info 2021-04-04T12:30:43.842 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool info 2021-04-04T12:30:43.880 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: image 2021-04-04T12:30:43.884 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirrorns2 mirror pool status 2021-04-04T12:30:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirrorns2 mirror pool status 2021-04-04T12:30:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool status --verbose 2021-04-04T12:30:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: UNKNOWN 2021-04-04T12:30:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: UNKNOWN 2021-04-04T12:30:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: OK 2021-04-04T12:30:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 0 total 2021-04-04T12:30:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2021-04-04T12:30:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: none 2021-04-04T12:30:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:43.961 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2021-04-04T12:30:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 ls 2021-04-04T12:30:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test info 2021-04-04T12:30:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 info test 2021-04-04T12:30:44.002 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test info 2021-04-04T12:30:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test': 2021-04-04T12:30:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 0 2021-04-04T12:30:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 1409c8b81b0c 2021-04-04T12:30:44.052 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.1409c8b81b0c 2021-04-04T12:30:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock 2021-04-04T12:30:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:44.053 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:40:47 2021 2021-04-04T12:30:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:40:47 2021 2021-04-04T12:30:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:40:47 2021 2021-04-04T12:30:44.056 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test journal status 2021-04-04T12:30:44.057 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test journal status 2021-04-04T12:30:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 journal status --image test 2021-04-04T12:30:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: journaling is not enabled for image test 2021-04-04T12:30:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:44.112 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test snapshots 2021-04-04T12:30:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test snapshots 2021-04-04T12:30:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 snap ls --all test 2021-04-04T12:30:44.172 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror ns2 rbd_mirroring omap vals 2021-04-04T12:30:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror ns2 rbd_mirroring omap vals 2021-04-04T12:30:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirroring 2021-04-04T12:30:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2021-04-04T12:30:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2021-04-04T12:30:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 00 00 00 |....| 2021-04-04T12:30:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2021-04-04T12:30:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2021-04-04T12:30:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2021-04-04T12:30:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 31 34 35 32 63 36 65 36 2d 62 31 31 62 2d 34 39 |1452c6e6-b11b-49| 2021-04-04T12:30:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 38 62 2d 61 33 61 64 2d 36 61 39 32 66 61 65 35 |8b-a3ad-6a92fae5| 2021-04-04T12:30:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 66 38 30 62 |f80b| 2021-04-04T12:30:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2021-04-04T12:30:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_af2694d6-07b0-4e37-95e1-92663262de9a_4ecbcd27-d351-4450-9326-27a96f352493 2021-04-04T12:30:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (96 bytes) : 2021-04-04T12:30:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 eb 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 fd 52 a0 c9 10 00 |...........R....| 2021-04-04T12:30:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 28 00 00 00 01 1a 00 00 00 65 72 72 |....(........err| 2021-04-04T12:30:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6f 72 20 62 6f 6f 74 73 74 72 61 70 70 69 6e 67 |or bootstrapping| 2021-04-04T12:30:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 20 72 65 70 6c 61 79 ab 89 69 60 d2 3e 71 0c 00 | replay..i`.>q..| 2021-04-04T12:30:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2021-04-04T12:30:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent images 2021-04-04T12:30:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parent images 2021-04-04T12:30:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ls -l 2021-04-04T12:30:44.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:NAME SIZE PARENT FMT PROT LOCK 2021-04-04T12:30:44.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_parent 128 MiB 2 2021-04-04T12:30:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_parent@snap 128 MiB 2 yes 2021-04-04T12:30:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent mirror pool info 2021-04-04T12:30:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parent mirror pool info 2021-04-04T12:30:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace mirror pool info 2021-04-04T12:30:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mode: image 2021-04-04T12:30:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Site Name: cluster2 2021-04-04T12:30:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Peer Sites: 2021-04-04T12:30:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:UUID: 66f7b743-d0ed-4180-9599-aa4c8b596b8d 2021-04-04T12:30:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Name: cluster1 2021-04-04T12:30:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Mirror UUID: 1cd21fe3-be3b-4a58-ac2d-a25de318debc 2021-04-04T12:30:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Direction: rx-tx 2021-04-04T12:30:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:Client: client.mirror 2021-04-04T12:30:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parent mirror pool status 2021-04-04T12:30:44.517 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent mirror pool status 2021-04-04T12:30:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror_parent --namespace mirror pool status --verbose 2021-04-04T12:30:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:health: UNKNOWN 2021-04-04T12:30:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:daemon health: UNKNOWN 2021-04-04T12:30:44.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image health: OK 2021-04-04T12:30:44.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:images: 1 total 2021-04-04T12:30:44.558 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 1 replaying 2021-04-04T12:30:44.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.559 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:DAEMONS 2021-04-04T12:30:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: none 2021-04-04T12:30:44.560 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.580 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:IMAGES 2021-04-04T12:30:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:test_parent: 2021-04-04T12:30:44.581 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: global_id: ed3379f2-3a26-4e23-8dd8-e2567a8361ce 2021-04-04T12:30:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: down+unknown 2021-04-04T12:30:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: status not found 2021-04-04T12:30:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04T12:30:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: peer_sites: 2021-04-04T12:30:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: name: cluster1 2021-04-04T12:30:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: state: up+replaying 2021-04-04T12:30:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: description: replaying, {"bytes_per_second":0.0,"entries_behind_primary":0,"entries_per_second":0.0,"non_primary_position":{"entry_tid":1,"object_number":1,"tag_tid":8},"primary_position":{"entry_tid":1,"object_number":1,"tag_tid":8}} 2021-04-04T12:30:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: last_update: 2021-04-04 12:30:29 2021-04-04T12:30:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ls 2021-04-04T12:30:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_parent info 2021-04-04T12:30:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_parent info 2021-04-04T12:30:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace info test_parent 2021-04-04T12:30:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:rbd image 'test_parent': 2021-04-04T12:30:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: size 128 MiB in 32 objects 2021-04-04T12:30:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: order 22 (4 MiB objects) 2021-04-04T12:30:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: snapshot_count: 1 2021-04-04T12:30:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: id: 121aa1d3d42f 2021-04-04T12:30:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: block_name_prefix: rbd_data.121aa1d3d42f 2021-04-04T12:30:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: format: 2 2021-04-04T12:30:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: features: layering, exclusive-lock, journaling 2021-04-04T12:30:44.686 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: op_features: 2021-04-04T12:30:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: flags: 2021-04-04T12:30:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: create_timestamp: Sun Apr 4 09:33:55 2021 2021-04-04T12:30:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: access_timestamp: Sun Apr 4 09:33:55 2021 2021-04-04T12:30:44.687 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: modify_timestamp: Sun Apr 4 09:33:55 2021 2021-04-04T12:30:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: journal: 121aa1d3d42f 2021-04-04T12:30:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring state: enabled 2021-04-04T12:30:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring mode: journal 2021-04-04T12:30:44.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring global id: ed3379f2-3a26-4e23-8dd8-e2567a8361ce 2021-04-04T12:30:44.689 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: mirroring primary: true 2021-04-04T12:30:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_parent journal status 2021-04-04T12:30:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_parent journal status 2021-04-04T12:30:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace journal status --image test_parent 2021-04-04T12:30:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:minimum_set: 0 2021-04-04T12:30:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:active_set: 0 2021-04-04T12:30:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:registered clients: 2021-04-04T12:30:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=, commit_position=[positions=[[object_number=1, tag_tid=8, entry_tid=1], [object_number=0, tag_tid=8, entry_tid=0], [object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2]]], state=connected] 2021-04-04T12:30:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: [id=1cd21fe3-be3b-4a58-ac2d-a25de318debc, commit_position=[positions=[[object_number=1, tag_tid=8, entry_tid=1], [object_number=0, tag_tid=8, entry_tid=0], [object_number=3, tag_tid=4, entry_tid=3], [object_number=2, tag_tid=4, entry_tid=2]]], state=connected] 2021-04-04T12:30:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image test_parent snapshots 2021-04-04T12:30:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo image test_parent snapshots 2021-04-04T12:30:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace snap ls --all test_parent 2021-04-04T12:30:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 2021-04-04T12:30:44.812 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 4 snap 128 MiB yes Sun Apr 4 09:33:55 2021 user 2021-04-04T12:30:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent rbd_mirroring omap vals 2021-04-04T12:30:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parent rbd_mirroring omap vals 2021-04-04T12:30:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace listomapvals rbd_mirroring 2021-04-04T12:30:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:global_ed3379f2-3a26-4e23-8dd8-e2567a8361ce 2021-04-04T12:30:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (16 bytes) : 2021-04-04T12:30:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 0c 00 00 00 31 32 31 61 61 31 64 33 64 34 32 66 |....121aa1d3d42f| 2021-04-04T12:30:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 2021-04-04T12:30:44.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:image_121aa1d3d42f 2021-04-04T12:30:44.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (48 bytes) : 2021-04-04T12:30:44.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 2a 00 00 00 24 00 00 00 65 64 33 33 37 39 |..*...$...ed3379| 2021-04-04T12:30:44.858 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 66 32 2d 33 61 32 36 2d 34 65 32 33 2d 38 64 64 |f2-3a26-4e23-8dd| 2021-04-04T12:30:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 38 2d 65 32 35 36 37 61 38 33 36 31 63 65 01 00 |8-e2567a8361ce..| 2021-04-04T12:30:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 2021-04-04T12:30:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_mode 2021-04-04T12:30:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (4 bytes) : 2021-04-04T12:30:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 00 00 00 |....| 2021-04-04T12:30:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000004 2021-04-04T12:30:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_peer_66f7b743-d0ed-4180-9599-aa4c8b596b8d 2021-04-04T12:30:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (132 bytes) : 2021-04-04T12:30:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 36 36 66 37 62 37 |..~...$...66f7b7| 2021-04-04T12:30:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 34 33 2d 64 30 65 64 2d 34 31 38 30 2d 39 35 39 |43-d0ed-4180-959| 2021-04-04T12:30:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 39 2d 61 61 34 63 38 62 35 39 36 62 38 64 08 00 |9-aa4c8b596b8d..| 2021-04-04T12:30:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 63 6c 75 73 74 65 72 31 0d 00 00 00 63 6c |..cluster1....cl| 2021-04-04T12:30:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2021-04-04T12:30:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 ff ff ff 02 24 00 00 00 31 63 64 32 31 66 65 33 |....$...1cd21fe3| 2021-04-04T12:30:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 2d 62 65 33 62 2d 34 61 35 38 2d 61 63 32 64 2d |-be3b-4a58-ac2d-| 2021-04-04T12:30:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 61 32 35 64 65 33 31 38 64 65 62 63 67 b1 69 60 |a25de318debcg.i`| 2021-04-04T12:30:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 ac bc a9 0a |....| 2021-04-04T12:30:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000084 2021-04-04T12:30:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:mirror_uuid 2021-04-04T12:30:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (36 bytes) : 2021-04-04T12:30:44.865 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 64 65 35 33 36 36 32 39 2d 65 62 31 65 2d 34 64 |de536629-eb1e-4d| 2021-04-04T12:30:44.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 34 30 2d 62 61 64 33 2d 37 62 62 37 62 32 62 34 |40-bad3-7bb7b2b4| 2021-04-04T12:30:44.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 36 33 66 64 |63fd| 2021-04-04T12:30:44.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000024 2021-04-04T12:30:44.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.866 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:remote_status_global_ed3379f2-3a26-4e23-8dd8-e2567a8361ce_1cd21fe3-be3b-4a58-ac2d-a25de318debc 2021-04-04T12:30:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:value (291 bytes) : 2021-04-04T12:30:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000000 01 01 2c 00 00 00 08 f7 13 00 00 00 00 00 00 01 |..,.............| 2021-04-04T12:30:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000010 01 01 1c 00 00 00 03 00 00 00 d5 45 04 51 10 00 |...........E.Q..| 2021-04-04T12:30:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000020 00 00 02 00 00 00 ac 15 0f 7a 00 00 00 00 00 00 |.........z......| 2021-04-04T12:30:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000030 00 00 01 01 eb 00 00 00 04 dd 00 00 00 72 65 70 |.............rep| 2021-04-04T12:30:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000040 6c 61 79 69 6e 67 2c 20 7b 22 62 79 74 65 73 5f |laying, {"bytes_| 2021-04-04T12:30:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000050 70 65 72 5f 73 65 63 6f 6e 64 22 3a 30 2e 30 2c |per_second":0.0,| 2021-04-04T12:30:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000060 22 65 6e 74 72 69 65 73 5f 62 65 68 69 6e 64 5f |"entries_behind_| 2021-04-04T12:30:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000070 70 72 69 6d 61 72 79 22 3a 30 2c 22 65 6e 74 72 |primary":0,"entr| 2021-04-04T12:30:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000080 69 65 73 5f 70 65 72 5f 73 65 63 6f 6e 64 22 3a |ies_per_second":| 2021-04-04T12:30:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000090 30 2e 30 2c 22 6e 6f 6e 5f 70 72 69 6d 61 72 79 |0.0,"non_primary| 2021-04-04T12:30:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000a0 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e 74 |_position":{"ent| 2021-04-04T12:30:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000b0 72 79 5f 74 69 64 22 3a 31 2c 22 6f 62 6a 65 63 |ry_tid":1,"objec| 2021-04-04T12:30:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000c0 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 61 67 |t_number":1,"tag| 2021-04-04T12:30:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000d0 5f 74 69 64 22 3a 38 7d 2c 22 70 72 69 6d 61 72 |_tid":8},"primar| 2021-04-04T12:30:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000e0 79 5f 70 6f 73 69 74 69 6f 6e 22 3a 7b 22 65 6e |y_position":{"en| 2021-04-04T12:30:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:000000f0 74 72 79 5f 74 69 64 22 3a 31 2c 22 6f 62 6a 65 |try_tid":1,"obje| 2021-04-04T12:30:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000100 63 74 5f 6e 75 6d 62 65 72 22 3a 31 2c 22 74 61 |ct_number":1,"ta| 2021-04-04T12:30:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000110 67 5f 74 69 64 22 3a 38 7d 7d 65 b1 69 60 ca df |g_tid":8}}e.i`..| 2021-04-04T12:30:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000120 e0 12 00 |...| 2021-04-04T12:30:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:00000123 2021-04-04T12:30:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent ns1 images 2021-04-04T12:30:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parent ns1 images 2021-04-04T12:30:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls -l 2021-04-04T12:30:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns1' does not exist. 2021-04-04T12:30:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: listing images failed: (2) No such file or directory 2021-04-04T12:30:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parentns1 mirror pool info 2021-04-04T12:30:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parentns1 mirror pool info 2021-04-04T12:30:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool info 2021-04-04T12:30:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns1' does not exist. 2021-04-04T12:30:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parentns1 mirror pool status 2021-04-04T12:30:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parentns1 mirror pool status 2021-04-04T12:30:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool status --verbose 2021-04-04T12:30:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns1' does not exist. 2021-04-04T12:30:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:44.989 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:44.990 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls 2021-04-04T12:30:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent ns1 rbd_mirroring omap vals 2021-04-04T12:30:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parent ns1 rbd_mirroring omap vals 2021-04-04T12:30:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2021-04-04T12:30:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2021-04-04T12:30:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:45.065 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:45.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent ns2 images 2021-04-04T12:30:45.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parent ns2 images 2021-04-04T12:30:45.066 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls -l 2021-04-04T12:30:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns2' does not exist. 2021-04-04T12:30:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: listing images failed: (2) No such file or directory 2021-04-04T12:30:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parentns2 mirror pool info 2021-04-04T12:30:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parentns2 mirror pool info 2021-04-04T12:30:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool info 2021-04-04T12:30:45.141 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns2' does not exist. 2021-04-04T12:30:45.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:45.144 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parentns2 mirror pool status 2021-04-04T12:30:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parentns2 mirror pool status 2021-04-04T12:30:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ CEPH_ARGS= rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool status --verbose 2021-04-04T12:30:45.182 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:rbd: namespace 'ns2' does not exist. 2021-04-04T12:30:45.185 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls 2021-04-04T12:30:45.225 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster2 mirror_parent ns2 rbd_mirroring omap vals 2021-04-04T12:30:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster2 mirror_parent ns2 rbd_mirroring omap vals 2021-04-04T12:30:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2021-04-04T12:30:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2021-04-04T12:30:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local ret 2021-04-04T12:30:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ seq 0 3 2021-04-04T12:30:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:0 2021-04-04T12:30:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:0 2021-04-04T12:30:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T12:30:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 cluster instance 2021-04-04T12:30:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T12:30:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T12:30:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T12:30:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T12:30:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T12:30:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T12:30:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T12:30:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T12:30:45.275 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T12:30:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T12:30:45.276 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 pid file 2021-04-04T12:30:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2021-04-04T12:30:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2021-04-04T12:30:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ ! -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid ] 2021-04-04T12:30:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid 2021-04-04T12:30:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2021-04-04T12:30:45.289 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pid=34123 2021-04-04T12:30:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -z 34123 ] 2021-04-04T12:30:45.290 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: rbd-mirror process in ps output: 2021-04-04T12:30:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo rbd-mirror process in ps output: 2021-04-04T12:30:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T12:30:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=34123 NR == 1 {print} $2 == pid {print; exit 1} 2021-04-04T12:30:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 2021-04-04T12:30:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:ubuntu 34123 0.8 0.3 1043944 104600 ? Ssl 09:40 1:28 rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T12:30:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:0 cluster1 2021-04-04T12:30:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:0 2021-04-04T12:30:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T12:30:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T12:30:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2021-04-04T12:30:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:0 2021-04-04T12:30:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T12:30:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T12:30:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T12:30:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T12:30:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 0 = cluster1:0 ] 2021-04-04T12:30:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T12:30:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T12:30:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=0 2021-04-04T12:30:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=0 2021-04-04T12:30:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2021-04-04T12:30:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T12:30:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2021-04-04T12:30:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ ! -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok ] 2021-04-04T12:30:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 rbd-mirror status 2021-04-04T12:30:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 rbd-mirror status 2021-04-04T12:30:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status 2021-04-04T12:30:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:{ 2021-04-04T12:30:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool_replayers": [ 2021-04-04T12:30:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T12:30:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "peer": "uuid: ba5e7ea1-e930-4550-b570-945e322d5ab4 cluster: cluster2 client: client.mirror", 2021-04-04T12:30:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool": "mirror", 2021-04-04T12:30:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4944", 2021-04-04T12:30:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "running", 2021-04-04T12:30:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader_instance_id": "4944", 2021-04-04T12:30:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader": true, 2021-04-04T12:30:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instances": [ 2021-04-04T12:30:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4944", 2021-04-04T12:30:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4948", 2021-04-04T12:30:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4957", 2021-04-04T12:30:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4961" 2021-04-04T12:30:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ], 2021-04-04T12:30:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.0.34123.cluster1.93932179281216.asok", 2021-04-04T12:30:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.34123.cluster2.93932191495488.asok", 2021-04-04T12:30:45.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "sync_throttler": { 2021-04-04T12:30:45.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 5, 2021-04-04T12:30:45.442 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T12:30:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T12:30:45.443 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "deletion_throttler": { 2021-04-04T12:30:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 1, 2021-04-04T12:30:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T12:30:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T12:30:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [ 2021-04-04T12:30:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T12:30:45.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "mirror/test_force_promote", 2021-04-04T12:30:45.447 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Stopped" 2021-04-04T12:30:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T12:30:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "mirror/test2", 2021-04-04T12:30:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T12:30:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T12:30:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "mirror/test_clone_v1", 2021-04-04T12:30:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T12:30:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T12:30:45.452 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "mirror/test1", 2021-04-04T12:30:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T12:30:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ], 2021-04-04T12:30:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter": { 2021-04-04T12:30:45.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter_status": { 2021-04-04T12:30:45.457 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "delete_images_queue": [], 2021-04-04T12:30:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "failed_deletes_queue": [] 2021-04-04T12:30:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "namespaces": [ 2021-04-04T12:30:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T12:30:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "ns1", 2021-04-04T12:30:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [], 2021-04-04T12:30:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter": { 2021-04-04T12:30:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter_status": { 2021-04-04T12:30:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "delete_images_queue": [], 2021-04-04T12:30:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "failed_deletes_queue": [] 2021-04-04T12:30:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T12:30:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "ns2", 2021-04-04T12:30:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [], 2021-04-04T12:30:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter": { 2021-04-04T12:30:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter_status": { 2021-04-04T12:30:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "delete_images_queue": [], 2021-04-04T12:30:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "failed_deletes_queue": [] 2021-04-04T12:30:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ] 2021-04-04T12:30:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T12:30:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "peer": "uuid: 062cbc91-4713-4ded-b744-6fda24ecf93a cluster: cluster2 client: client.mirror", 2021-04-04T12:30:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "pool": "mirror_parent", 2021-04-04T12:30:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instance_id": "4964", 2021-04-04T12:30:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "running", 2021-04-04T12:30:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader_instance_id": "4964", 2021-04-04T12:30:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "leader": true, 2021-04-04T12:30:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "instances": [ 2021-04-04T12:30:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4964", 2021-04-04T12:30:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4971", 2021-04-04T12:30:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "4975" 2021-04-04T12:30:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ], 2021-04-04T12:30:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "local_cluster_admin_socket": "/var/run/ceph/client.mirror.0.34123.cluster1.93932204488000.asok", 2021-04-04T12:30:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "remote_cluster_admin_socket": "/var/run/ceph/client.mirror.34123.cluster2.93932216907072.asok", 2021-04-04T12:30:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "sync_throttler": { 2021-04-04T12:30:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 5, 2021-04-04T12:30:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T12:30:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T12:30:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "deletion_throttler": { 2021-04-04T12:30:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "max_parallel_requests": 1, 2021-04-04T12:30:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "running_requests": 0, 2021-04-04T12:30:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "waiting_requests": 0 2021-04-04T12:30:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_replayers": [ 2021-04-04T12:30:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: { 2021-04-04T12:30:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "name": "mirror_parent/test_parent", 2021-04-04T12:30:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "state": "Replaying" 2021-04-04T12:30:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ], 2021-04-04T12:30:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter": { 2021-04-04T12:30:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "image_deleter_status": { 2021-04-04T12:30:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "delete_images_queue": [], 2021-04-04T12:30:45.477 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "failed_deletes_queue": [] 2021-04-04T12:30:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: }, 2021-04-04T12:30:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: "namespaces": [] 2021-04-04T12:30:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: } 2021-04-04T12:30:45.478 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: ] 2021-04-04T12:30:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:} 2021-04-04T12:30:45.479 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:45.483 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_pid_file cluster1:1 2021-04-04T12:30:45.484 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1:1 2021-04-04T12:30:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T12:30:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 cluster instance 2021-04-04T12:30:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T12:30:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=cluster 2021-04-04T12:30:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T12:30:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T12:30:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T12:30:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T12:30:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval cluster=cluster1 2021-04-04T12:30:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T12:30:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T12:30:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T12:30:45.489 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 pid file 2021-04-04T12:30:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2021-04-04T12:30:45.490 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2021-04-04T12:30:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ ! -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid ] 2021-04-04T12:30:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local pid 2021-04-04T12:30:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2021-04-04T12:30:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ pid=34151 2021-04-04T12:30:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ -z 34151 ] 2021-04-04T12:30:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo rbd-mirror process in ps output: 2021-04-04T12:30:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: rbd-mirror process in ps output: 2021-04-04T12:30:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ps auxww 2021-04-04T12:30:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ awk -v pid=34151 NR == 1 {print} $2 == pid {print; exit 1} 2021-04-04T12:30:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 2021-04-04T12:30:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:ubuntu 34151 0.1 0.2 966096 92888 ? Ssl 09:40 0:17 rbd-mirror --cluster cluster1 --id mirror.1 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2021-04-04T12:30:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo 2021-04-04T12:30:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout: 2021-04-04T12:30:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ daemon_asok_file cluster1:1 cluster1 2021-04-04T12:30:45.500 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local local_cluster=cluster1:1 2021-04-04T12:30:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster=cluster1 2021-04-04T12:30:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance 2021-04-04T12:30:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2021-04-04T12:30:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local val=cluster1:1 2021-04-04T12:30:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local cluster_var_name=local_cluster 2021-04-04T12:30:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local instance_var_name=instance 2021-04-04T12:30:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ cluster=cluster1 2021-04-04T12:30:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T12:30:45.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ 1 = cluster1:1 ] 2021-04-04T12:30:45.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval local_cluster=cluster1 2021-04-04T12:30:45.504 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local_cluster=cluster1 2021-04-04T12:30:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ eval instance=1 2021-04-04T12:30:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ instance=1 2021-04-04T12:30:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2021-04-04T12:30:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T12:30:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2021-04-04T12:30:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ [ ! -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok ] 2021-04-04T12:30:45.508 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stdout:cluster1 rbd-mirror status 2021-04-04T12:30:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ echo cluster1 rbd-mirror status 2021-04-04T12:30:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi122.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status 2021-04-04T12:48:41.910 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T12:48:41.906+0000 7f6a80fb6700 -1 reset not still connected to 0x555cac97ce50 2021-04-04T12:48:41.949 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T12:48:41.906+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89dad0 2021-04-04T12:54:03.198 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T12:54:03.196+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4eab10 2021-04-04T12:54:03.199 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T12:54:03.196+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad7f7a80 2021-04-04T12:54:03.199 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T12:54:03.196+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad9ba8a0 2021-04-04T12:54:03.199 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T12:54:03.196+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89e7d0 2021-04-04T12:54:03.200 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T12:54:03.196+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89e8a0 2021-04-04T13:15:29.226 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T13:15:29.222+0000 7f6a80fb6700 -1 reset not still connected to 0x555cac97d5a0 2021-04-04T13:16:32.171 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T13:16:32.167+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f7283c0 2021-04-04T13:25:59.544 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T13:25:59.539+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c410 2021-04-04T13:25:59.545 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T13:25:59.539+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c750 2021-04-04T13:42:04.011 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T13:42:04.007+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89dad0 2021-04-04T13:50:45.850 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T13:50:45.846+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c410 2021-04-04T13:50:45.874 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T13:50:45.846+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c750 2021-04-04T13:55:34.586 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T13:55:34.585+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ea080 2021-04-04T13:55:34.587 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T13:55:34.585+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ebcf0 2021-04-04T13:55:34.587 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T13:55:34.585+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c410 2021-04-04T13:55:34.587 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T13:55:34.585+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c750 2021-04-04T14:10:41.778 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T14:10:41.777+0000 7f6a80fb6700 -1 reset not still connected to 0x555cac97d5a0 2021-04-04T14:27:05.057 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T14:27:05.052+0000 7f850200c700 -1 reset not still connected to 0x5605c1720f70 2021-04-04T14:27:05.058 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T14:27:05.052+0000 7f850200c700 -1 reset not still connected to 0x5605c1723e90 2021-04-04T14:33:22.834 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T14:33:22.829+0000 7f4337ec2700 -1 reset not still connected to 0x55966009b740 2021-04-04T14:33:22.835 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T14:33:22.829+0000 7f4337ec2700 -1 reset not still connected to 0x55966009b810 2021-04-04T14:33:22.835 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T14:33:22.829+0000 7f4337ec2700 -1 reset not still connected to 0x559661094cb0 2021-04-04T14:33:22.835 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T14:33:22.829+0000 7f4337ec2700 -1 reset not still connected to 0x5596616430c0 2021-04-04T14:43:17.662 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T14:43:17.653+0000 7f850200c700 -1 reset not still connected to 0x5605c1720f70 2021-04-04T14:43:17.688 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T14:43:17.653+0000 7f850200c700 -1 reset not still connected to 0x5605c1723e90 2021-04-04T14:53:53.532 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T14:53:53.527+0000 7f850200c700 -1 reset not still connected to 0x5605c1de54d0 2021-04-04T15:05:12.282 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T15:05:12.275+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c750 2021-04-04T15:22:45.140 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T15:22:45.137+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c410 2021-04-04T15:22:45.141 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T15:22:45.137+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c750 2021-04-04T15:29:57.951 INFO:tasks.ceph.cluster2.osd.1.smithi122.stderr:2021-04-04T15:29:57.948+0000 7f6f409ce700 -1 reset not still connected to 0x55d2c8955a80 2021-04-04T15:53:34.893 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T15:53:34.887+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f7283c0 2021-04-04T15:57:34.960 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T15:57:34.953+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f728cb0 2021-04-04T16:08:35.130 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T16:08:35.122+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f728cb0 2021-04-04T16:17:02.673 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:17:02.668+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4eab10 2021-04-04T16:17:02.674 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:17:02.668+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad7f7a80 2021-04-04T16:17:02.675 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:17:02.668+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad9ba8a0 2021-04-04T16:17:02.675 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:17:02.668+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89e7d0 2021-04-04T16:17:02.675 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:17:02.668+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89e8a0 2021-04-04T16:21:19.822 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:21:19.816+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ea080 2021-04-04T16:21:19.823 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:21:19.816+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ebcf0 2021-04-04T16:21:19.824 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:21:19.816+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c410 2021-04-04T16:21:19.824 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:21:19.816+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c750 2021-04-04T16:24:43.029 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:24:43.024+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4eab10 2021-04-04T16:24:43.030 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:24:43.024+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad7f7a80 2021-04-04T16:24:43.030 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:24:43.024+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad9ba8a0 2021-04-04T16:24:43.030 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:24:43.024+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89e7d0 2021-04-04T16:24:43.031 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:24:43.024+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89e8a0 2021-04-04T16:28:41.499 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T16:28:41.494+0000 7f850200c700 -1 reset not still connected to 0x5605c1720f70 2021-04-04T16:28:41.500 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T16:28:41.494+0000 7f850200c700 -1 reset not still connected to 0x5605c1723e90 2021-04-04T16:43:41.832 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T16:43:41.826+0000 7f850200c700 -1 reset not still connected to 0x5605c1720f70 2021-04-04T16:43:41.857 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T16:43:41.826+0000 7f850200c700 -1 reset not still connected to 0x5605c1723e90 2021-04-04T16:55:49.881 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:55:49.876+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ea080 2021-04-04T16:55:49.881 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:55:49.876+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ebcf0 2021-04-04T16:55:49.882 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:55:49.876+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c410 2021-04-04T16:55:49.882 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T16:55:49.876+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c750 2021-04-04T17:09:29.288 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T17:09:29.278+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ea080 2021-04-04T17:09:29.334 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T17:09:29.278+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ebcf0 2021-04-04T17:09:29.334 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T17:09:29.278+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c410 2021-04-04T17:09:29.334 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T17:09:29.278+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c750 2021-04-04T17:15:27.130 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T17:15:27.122+0000 7f850200c700 -1 reset not still connected to 0x5605c1720f70 2021-04-04T17:15:27.131 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T17:15:27.122+0000 7f850200c700 -1 reset not still connected to 0x5605c1723e90 2021-04-04T17:19:10.572 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T17:19:10.564+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ea080 2021-04-04T17:19:10.602 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T17:19:10.564+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ebcf0 2021-04-04T17:19:10.603 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T17:19:10.568+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c410 2021-04-04T17:19:10.604 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T17:19:10.568+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c750 2021-04-04T17:52:09.039 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T17:52:09.030+0000 7f6a80fb6700 -1 reset not still connected to 0x555cac97ce50 2021-04-04T17:52:09.068 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T17:52:09.030+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89dad0 2021-04-04T17:58:57.431 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T17:58:57.423+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c410 2021-04-04T17:58:57.432 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T17:58:57.423+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c750 2021-04-04T17:59:22.455 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T17:59:22.447+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4eab10 2021-04-04T17:59:22.456 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T17:59:22.447+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89e7d0 2021-04-04T17:59:22.456 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T17:59:22.447+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89e8a0 2021-04-04T18:06:37.790 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T18:06:37.779+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89e7d0 2021-04-04T18:06:37.814 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T18:06:37.783+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89e8a0 2021-04-04T18:13:30.005 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T18:13:30.000+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ea080 2021-04-04T18:13:30.006 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T18:13:30.000+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ebcf0 2021-04-04T18:13:30.006 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T18:13:30.000+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c410 2021-04-04T18:13:30.006 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T18:13:30.000+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c750 2021-04-04T18:16:06.243 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T18:16:06.237+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47ed31810 2021-04-04T18:16:06.244 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T18:16:06.237+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f728ff0 2021-04-04T18:16:06.244 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T18:16:06.237+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f729c20 2021-04-04T18:35:40.038 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T18:35:40.030+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47ed31810 2021-04-04T18:35:40.067 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T18:35:40.030+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f728ff0 2021-04-04T18:35:40.067 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T18:35:40.030+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f729c20 2021-04-04T18:36:25.118 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T18:36:25.113+0000 7f850200c700 -1 reset not still connected to 0x5605c1720f70 2021-04-04T18:36:25.118 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T18:36:25.113+0000 7f850200c700 -1 reset not still connected to 0x5605c1723e90 2021-04-04T18:43:05.051 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T18:43:05.048+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47ed31810 2021-04-04T18:43:05.052 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T18:43:05.048+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f728ff0 2021-04-04T18:43:05.052 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T18:43:05.048+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f729c20 2021-04-04T19:47:25.159 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T19:47:25.153+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ea080 2021-04-04T19:47:25.192 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T19:47:25.153+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ebcf0 2021-04-04T19:47:25.192 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T19:47:25.153+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c410 2021-04-04T19:47:25.193 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T19:47:25.153+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c750 2021-04-04T19:48:17.532 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T19:48:17.529+0000 7f4337ec2700 -1 reset not still connected to 0x55966009b740 2021-04-04T19:48:17.532 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T19:48:17.529+0000 7f4337ec2700 -1 reset not still connected to 0x55966009b810 2021-04-04T19:48:17.532 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T19:48:17.529+0000 7f4337ec2700 -1 reset not still connected to 0x559661094cb0 2021-04-04T19:48:17.533 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T19:48:17.529+0000 7f4337ec2700 -1 reset not still connected to 0x5596616430c0 2021-04-04T19:51:10.167 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T19:51:10.158+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ea080 2021-04-04T19:51:10.168 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T19:51:10.158+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ebcf0 2021-04-04T19:51:10.168 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T19:51:10.158+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c410 2021-04-04T19:51:10.168 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T19:51:10.158+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c750 2021-04-04T20:05:05.190 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T20:05:05.182+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47ed31810 2021-04-04T20:08:43.489 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T20:08:43.484+0000 7f4337ec2700 -1 reset not still connected to 0x5596616430c0 2021-04-04T20:15:10.206 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T20:15:10.197+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ea080 2021-04-04T20:15:10.207 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T20:15:10.197+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ebcf0 2021-04-04T20:15:10.207 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T20:15:10.197+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c410 2021-04-04T20:15:10.207 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T20:15:10.197+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c750 2021-04-04T20:15:10.212 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T20:15:10.205+0000 7f6a80fb6700 -1 reset not still connected to 0x555cab748560 2021-04-04T20:15:10.212 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T20:15:10.205+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ea080 2021-04-04T20:15:10.213 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T20:15:10.205+0000 7f6a80fb6700 -1 reset not still connected to 0x555cad4ebcf0 2021-04-04T20:15:10.213 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T20:15:10.205+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c410 2021-04-04T20:15:10.213 INFO:tasks.ceph.cluster1.osd.2.smithi078.stderr:2021-04-04T20:15:10.205+0000 7f6a80fb6700 -1 reset not still connected to 0x555cae89c750 2021-04-04T20:20:54.050 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T20:20:54.048+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47e924cb0 2021-04-04T20:20:54.051 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T20:20:54.048+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f7261a0 2021-04-04T20:38:12.161 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T20:38:12.159+0000 7f850200c700 -1 reset not still connected to 0x5605c1720f70 2021-04-04T20:38:12.162 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T20:38:12.159+0000 7f850200c700 -1 reset not still connected to 0x5605c1723e90 2021-04-04T20:45:45.256 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T20:45:45.249+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47e503b50 2021-04-04T20:45:45.284 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T20:45:45.249+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47ed31810 2021-04-04T20:45:45.284 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T20:45:45.249+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f728ff0 2021-04-04T20:45:45.284 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T20:45:45.249+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f729c20 2021-04-04T20:50:05.411 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T20:50:05.401+0000 7f4337ec2700 -1 reset not still connected to 0x55966009b810 2021-04-04T20:50:05.412 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T20:50:05.405+0000 7f4337ec2700 -1 reset not still connected to 0x559661094cb0 2021-04-04T20:50:05.412 INFO:tasks.ceph.cluster1.osd.1.smithi078.stderr:2021-04-04T20:50:05.405+0000 7f4337ec2700 -1 reset not still connected to 0x5596616430c0 2021-04-04T21:00:48.190 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T21:00:48.184+0000 7f850200c700 -1 reset not still connected to 0x5605c1720f70 2021-04-04T21:00:48.191 INFO:tasks.ceph.cluster2.osd.0.smithi122.stderr:2021-04-04T21:00:48.184+0000 7f850200c700 -1 reset not still connected to 0x5605c1723e90 2021-04-04T21:08:09.785 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T21:08:09.778+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f7283c0 2021-04-04T21:11:20.301 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T21:11:20.293+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47ed31810 2021-04-04T21:11:20.302 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T21:11:20.293+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f728ff0 2021-04-04T21:11:20.302 INFO:tasks.ceph.cluster2.osd.2.smithi122.stderr:2021-04-04T21:11:20.293+0000 7fc4b9f07700 -1 reset not still connected to 0x55a47f729c20 2021-04-04T21:21:59.658 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-04-04T21:21:59.661 DEBUG:teuthology.task.console_log:Killing console logger for smithi078 2021-04-04T21:21:59.662 DEBUG:teuthology.task.console_log:Killing console logger for smithi122 2021-04-04T21:21:59.662 DEBUG:teuthology.exit:Finished running handlers